This is the first version patch to add an integration test for bins package. The test is following the steps in https://github.com/alexjg/linkd-playground. This version covers the steps before we git push the local changes and do git clone. This test is not embedded in CI workflow yet. Run `cargo test` (recommend `cargo test -- --nocapture`) under bins to execute the test. Han Xu (1): basic integration test for bins bins/Cargo.lock | 29 +++ bins/Cargo.toml | 1 + bins/tests/Cargo.toml | 13 ++ bins/tests/integration_test.rs | 392 +++++++++++++++++++++++++++++++++ 4 files changed, 435 insertions(+) create mode 100644 bins/tests/Cargo.toml create mode 100644 bins/tests/integration_test.rs -- 2.32.0 (Apple Git-132)
radicle-link/patches/nixos-latest.yml: FAILED in 23m57s [add integration test for bins package][0] from [Han Xu][1] [0]: https://lists.sr.ht/~radicle-link/dev/patches/34576 [1]: mailto:keepsimple@gmail.com ✗ #820143 FAILED radicle-link/patches/nixos-latest.yml https://builds.sr.ht/~radicle-link/job/820143
This is the 3rd version (re-rolling) patch to add an integration test for `bins` package. The test case scenario follows the steps in https://github.com/alexjg/linkd-playground , except it is in Rust now and we can just run `cargo test`. To see a more helpful output, please run `cargo test -- --nocapture` from the `bins/tests` directory. The only non-test code change is in git version checks. In our testing, it shows that git 2.25.1 on Ubuntu 20.04 and git 2.30.2 on Debian 11 do not need the workaround that adds "namespaces" manually. Hence fixing the git version checked against. This patch also addresses the comments from the 2nd version, and I opted to re-rolling instead of adding new commits. Han Xu (1): Add integration test for bins and fix git version checks bins/Cargo.lock | 29 +++ bins/Cargo.toml | 1 + bins/tests/Cargo.toml | 13 + bins/tests/integration_test.rs | 449 +++++++++++++++++++++++++++++++++ link-git/src/protocol/fetch.rs | 5 +- link-git/src/protocol/ls.rs | 5 +- 6 files changed, 500 insertions(+), 2 deletions(-) create mode 100644 bins/tests/Cargo.toml create mode 100644 bins/tests/integration_test.rs
Does this [...] mean something? (sorry I don't know)
This patch is an update on the previous v3 patch, with the following changes: - Added a check for refs/heads/master between peers. This is because I noticed a bug where `git push` from peer1 only pushs an older commit as `heads/master` to the seed node. - Found the root cause of the above bug and included a fix in cli/gitd-lib/src/hooks.rs. The root cause is that currently a peer does request_pull before updating the signed refs, hence the request pull only has an old signed refs for heads/master. Han Xu (3): Add integration test for bins and fix git version checks check peer1 and peer2 ref/heads/master fix post_receive to send updated refs in requst_pull bins/Cargo.lock | 29 ++ bins/Cargo.toml | 1 + bins/tests/Cargo.toml | 13 + bins/tests/integration_test.rs | 468 +++++++++++++++++++++++++++++++++ cli/gitd-lib/src/hooks.rs | 19 +- link-git/src/protocol/fetch.rs | 5 +- link-git/src/protocol/ls.rs | 5 +- 7 files changed, 530 insertions(+), 10 deletions(-) create mode 100644 bins/tests/Cargo.toml create mode 100644 bins/tests/integration_test.rs
This is the 5th version (re-rolling) patch to add an integration test for `bins` package. It is built on top of version 4 patch and addressed review comments (see new commits): - created a new macro to simplify the test code. - removed the use of systemd-socket-activate. The test case scenario follows the steps in https://github.com/alexjg/linkd-playground , except it is in Rust now and we can just run `cargo test`. To see a more helpful output, please run `cargo test -- --nocapture` from the `bins/tests` directory. We also found two bugs along the way: 1. Git version checks are off. In our testing, it shows that git 2.25.1 on Ubuntu 20.04 and git 2.30.2 on Debian 11 do not need the workaround that adds "namespaces" manually. Hence fixing the git version checked against. 2. `request_pull` to the seed fails to pull the latest commit. The reason being that the signed_refs for `heads/master` is not the latest. A fix is provided. Han Xu (5): Add integration test for bins and fix git version checks check peer1 and peer2 ref/heads/master fix post_receive to send updated refs in requst_pull Use macro to refactor run_lnk remove the use of systemd-socket-activate bins/Cargo.lock | 29 +++ bins/Cargo.toml | 1 + bins/tests/Cargo.toml | 13 + bins/tests/integration_test.rs | 417 +++++++++++++++++++++++++++++++++ cli/gitd-lib/src/hooks.rs | 19 +- link-git/src/protocol/fetch.rs | 5 +- link-git/src/protocol/ls.rs | 5 +- 7 files changed, 479 insertions(+), 10 deletions(-) create mode 100644 bins/tests/Cargo.toml create mode 100644 bins/tests/integration_test.rs
I still find this return and switch on the returned `bool` a bit cumbersome. Is it worth using something like https://docs.rs/pty-process/latest/pty_process/struct.Child.html which would give us basically the same API as `std::process::Child`? Failing that we could write a little macro to remove all the `is_parent` checks in the calling code.
So I am trying to get some collaboration on testing going between testing CLI type binaries across Is there a reason none of these could be used ?
I am open to use any of those if they are deemed necessary or helpful. So far I've built the integration test using the basic Rust / Cargo test facility and only one crate (`pty`). I felt it provides a good baseline for a testing of `bins`. We can always refactor the test code once we decide to use some of the crates you listed. Another thing is that, as you probably mentioned I believe, a well-documented test strategy is needed before we adopt some framework-like crates, for example instra. If we already have it, please let me know, I might have missed something. If we are in the process of defining the test strategy, I'm open to the discussion.
https://crates.io/crates/trycmd https://crates.io/crates/term-transcript out of date: https://github.com/rust-rspec/rspec https://crates.io/crates/trybuild https://nexte.st/book/leaky-tests.html compiletest seems to have detached yay https://github.com/Manishearth/compiletest-rs and ofc insta https://crates.io/crates/insta https://crates.io/crates/snapbox https://crates.io/crates/assert_cmd, https://crates.io/crates/assert_fs https://crates.io/crates/duct https://crates.io/crates/rexpect https://crates.io/crates/dir-diff I use insta a lot myself
Is there an example of how to use insta for testing some parts of radicle-link? An example will be good if we want to adopt it going forward.
Rust idiomatic way is to use small crates from the ecosystem If we are still going ahead of using Rust to do integration tests.
Which ones do you find useful/have tried? How would they be used to change over the code that Han is providing?
------- Original Message -------
This is the 6th version (re-rolling) patch to add an integration test for `bins` package. It is built on top of version 5 patch and addressed review comments: - refactor the code styling. - squash the patch into two commits only. The test case scenario follows the steps in https://github.com/alexjg/linkd-playground , except it is in Rust now and we can just run `cargo test`. To see a more helpful output, please run `cargo test -- --nocapture` from the `bins/tests` directory. We also found two bugs along the way: 1. Git version checks are off. In our testing, it shows that git 2.25.1 on Ubuntu 20.04 and git 2.30.2 on Debian 11 do not need the workaround that adds "namespaces" manually. Hence fixing the git version checked against. 2. `request_pull` to the seed fails to pull the latest commit. The reason being that the signed_refs for `heads/master` is not the latest. A fix is provided. Han Xu (2): Add integration test for bins and fix git version checks fix post_receive to send updated refs in request_pull bins/Cargo.lock | 29 +++ bins/Cargo.toml | 1 + bins/tests/Cargo.toml | 13 + bins/tests/integration_test.rs | 456 +++++++++++++++++++++++++++++++++ cli/gitd-lib/src/hooks.rs | 19 +- link-git/src/protocol/fetch.rs | 5 +- link-git/src/protocol/ls.rs | 5 +- 7 files changed, 518 insertions(+), 10 deletions(-) create mode 100644 bins/tests/Cargo.toml create mode 100644 bins/tests/integration_test.rs
This is the 9th version (re-rolling) patch to add an integration test for `bins` package. The only change from the 8th version is to add sign-off lines in commit messages. The test case scenario follows the steps in https://github.com/alexjg/linkd-playground , except it is in Rust now and we can just run `cargo test`. To see a more helpful output, please run `cargo test -- --nocapture` from the `bins/tests` directory. We also found two bugs along the way: 1. Git version checks are off. In our testing, it shows that git 2.25.1 on Ubuntu 20.04 and git 2.30.2 on Debian 11 do not need the workaround that adds "namespaces" manually. Hence fixing the git version check in ls-refs. 2. `request_pull` to the seed fails to pull the latest commit. The reason being that the signed_refs for `heads/master` is not the latest. A fix is provided. Han Xu (3): fix post_receive to send updated refs in request_pull Add integration test for bins fix Git version check for ls-refs bins/Cargo.lock | 29 +++ bins/Cargo.toml | 1 + bins/tests/Cargo.toml | 13 + bins/tests/integration_test.rs | 456 +++++++++++++++++++++++++++++++++ cli/gitd-lib/src/hooks.rs | 19 +- link-git/src/protocol/ls.rs | 5 +- 6 files changed, 514 insertions(+), 9 deletions(-) create mode 100644 bins/tests/Cargo.toml create mode 100644 bins/tests/integration_test.rs
This is the 8th version (re-rolling) patch to add an integration test for `bins` package. It was refactored the patch into 3 commits so that it is clearer to understand the changes. The test case scenario follows the steps in https://github.com/alexjg/linkd-playground , except it is in Rust now and we can just run `cargo test`. To see a more helpful output, please run `cargo test -- --nocapture` from the `bins/tests` directory. We also found two bugs along the way: 1. Git version checks are off. In our testing, it shows that git 2.25.1 on Ubuntu 20.04 and git 2.30.2 on Debian 11 do not need the workaround that adds "namespaces" manually. Hence fixing the git version check in ls-refs.
Wait, if Debian 11 needs git 2.30.2 then why would we lower the bounds to 2.25.1? If a Debian 11 computer was running this code and had 2.25.1, then it would say it doesn't need to explicitly namespace but it actually *does* until 2.30.2. So shouldn't the lower bound be 2.30.1 to maximise the usable OS's? Correct me if I'm wrong :)
2. `request_pull` to the seed fails to pull the latest commit. The reason being that the signed_refs for `heads/master` is not the latest. A fix is provided. Han Xu (3): fix post_receive to send updated refs in request_pull Add integration test for bins fix Git version check for ls-refs bins/Cargo.lock | 29 +++ bins/Cargo.toml | 1 + bins/tests/Cargo.toml | 13 + bins/tests/integration_test.rs | 456 +++++++++++++++++++++++++++++++++ cli/gitd-lib/src/hooks.rs | 19 +- link-git/src/protocol/ls.rs | 5 +- 6 files changed, 514 insertions(+), 9 deletions(-) create mode 100644 bins/tests/Cargo.toml create mode 100644 bins/tests/integration_test.rs
No, As a developer - It is our job to choose the right tool for the job - Re-using is one of the core competencies of ours - I don't see even attempts to evaluate the options I gave you.
Alex Good <alex@memoryandthought.me>To me that seems like work that you should be doing. Both Han and I have responded to a previous message you sent about using a library for this. We both took the time to outline our concerns in detail, for some reason you decided not to respond. It seems then that we have a difference of opinion on what should be done here. We're all collaborators here, and you seem to have the most experience with these libraries, you're also the person most interested in using these libraries, as such you seem best placed to present the case for them - I would be interested to hear it.
Don't expect anyone else to effectively do your work after for you.
Alex Good <alex@memoryandthought.me>This is an extremely uncharitable take. Everyone here is working hard in good faith. Han is on the fith reroll of this patch and has been very willing to accomodate feedback - he's clearly not expecting other people to do his work.
Correcting mistakes is one but not even looking at the list and engaging in technical merits is another. It is a massive stress having to clean up after someone when they were told of options to do it properly without extra effort. We don't really need a strategy to do the right thing (tm) This fallacy goes two ways - Now we are adding code that should not exists without strategy - We are now adding some low level code that is wholly redundant to test other low level code.. I mean what are we testing here.. the tests or the code ? Like why are we dealing with pty's and the logistics of it ? When e.g. trycmd - first on the list - does that in minimal manner ?
Alex Good <alex@memoryandthought.me>`trycmd` doesn't seem minimal to me, it requires rewriting your tests as literate markdown files. This is quite dramatic - maybe it's good, maybe it's bad, but it's certainly not an obvious decision. As I said, you seem to be more familiar with all these libraries, I would be keen to see what you think the code should look like and what you think the advantages and disadvantages are.
Think of it like this: Every line of code is liability. Just pick the right tool for the job (tm) ------- Original Message -------
On Mon, Aug 29, 2022 at 12:13 PM Fintan Halpenny <fintan.halpenny@gmail.com> wrote:
This is the 7th version (re-rolling) patch to add an integration test for `bins` package. It has been rebased on master branch 622c1bcd59a6ce584f957ffe6b874b2af0b207fd. The test case scenario follows the steps in https://github.com/alexjg/linkd-playground , except it is in Rust now and we can just run `cargo test`. To see a more helpful output, please run `cargo test -- --nocapture` from the `bins/tests` directory. We also found two bugs along the way: 1. Git version checks are off. In our testing, it shows that git 2.25.1 on Ubuntu 20.04 and git 2.30.2 on Debian 11 do not need the workaround that adds "namespaces" manually. Hence fixing the git version checked against. 2. `request_pull` to the seed fails to pull the latest commit. The reason being that the signed_refs for `heads/master` is not the latest. A fix is provided. Han Xu (2): Add integration test for bins and fix git version checks fix post_receive to send updated refs in request_pull bins/Cargo.lock | 29 +++ bins/Cargo.toml | 1 + bins/tests/Cargo.toml | 13 + bins/tests/integration_test.rs | 456 +++++++++++++++++++++++++++++++++ cli/gitd-lib/src/hooks.rs | 19 +- link-git/src/protocol/fetch.rs | 5 +- link-git/src/protocol/ls.rs | 5 +- 7 files changed, 518 insertions(+), 10 deletions(-) create mode 100644 bins/tests/Cargo.toml create mode 100644 bins/tests/integration_test.rs
Could you publish the v6 tag, please? :)
Here it is: https://github.com/keepsimple1/radicle-link/releases/tag/patches%2Fintegration-test-bins%2Fv6 Sorry I forgot it. Han
No worries, thanks for sending it on :) There's some unrelated (I believe) tests failing when I apply this patch on top of the latest master. I'm not sure why this is happening so hesitant to merge it just yet. Could you rebase off origin/master and see if you see the same behaviour running `scripts/ci/run`?
On Tue, Aug 30, 2022 at 1:50 AM Fintan Halpenny <fintan.halpenny@gmail.com> wrote:
On Tue, Aug 30, 2022 at 9:38 AM Fintan Halpenny <fintan.halpenny@gmail.com> wrote:
Could you publish your tag? :D
Sorry, I always forgot to publish the tag. Here it is: https://github.com/keepsimple1/radicle-link/releases/tag/patches%2Fintegration-test-bins%2Fv7 On Thu, Sep 1, 2022 at 2:38 AM Fintan Halpenny <fintan.halpenny@gmail.com> wrote:
We have changed the Git version checks in both link-git/src/protocol/ls.rs and link-git/src/protocol/fetch.rs. I reverted the fetch.rs change and now link-git tests are passing, without updating Git in the system. (My system has Git 2.25.1). Our new Integration Test uses `ls.rs` to do `ls-refs` to obtain refs path, so that without the fetch.rs change, Integration Test still passes. Given these observations, I am changing only the `ls.rs` for Git version check. I will submit a new patch version, and separate out the Git version check in its own commit so it is easier to see. thanks Han On Thu, Sep 1, 2022 at 4:23 AM Fintan Halpenny <fintan.halpenny@gmail.com> wrote:
I have submitted the new (8th) version of the patch, also published the tag : https://github.com/keepsimple1/radicle-link/releases/tag/patches%2Fintegration-test-bins%2Fv8 Cheers, Han
Also, I noticed that your commits are missing your sign-off, ie. name and email, as per our DCO policy[0]. [0]: https://github.com/radicle-dev/radicle-link/blob/master/CONTRIBUTING.md#certificate-of-origin
Will do. Thanks for the pointer.
On Mon, Sep 5, 2022 at 12:33 AM Fintan Halpenny <fintan.halpenny@gmail.com> wrote:
On Mon, Sep 5, 2022 at 12:55 AM Fintan Halpenny <fintan.halpenny@gmail.com> wrote:
LGTM! Can you publish the tag? :)
sorry I never remember to publish a tag ;-) Here it is: https://github.com/keepsimple1/radicle-link/releases/tag/patches%2Fintegration-test-bins%2Fv9 On Tue, Sep 6, 2022 at 2:24 AM Fintan Halpenny <fintan.halpenny@gmail.com> wrote:
Copy & paste the following snippet into your terminal to import this patchset into git:
curl -s https://lists.sr.ht/~radicle-link/dev/patches/34576/mbox | git am -3Learn more about email & git
I have some stylistic suggestions for test file below. I think you can also squash the changes to that same file into a single commit. So, the patch should look like the addition of the integration_test.rs file and the fix to the signed_refs update.
Thanks Fintan for your review. I've updated the patch to adopt your stylistic suggestions. And also squashed all changes into two commits like you said. Let me know if you have any questions. The 6th version is sent. Han
--- bins/tests/integration_test.rs | 165 ++++++++++++++++++++------------- 1 file changed, 100 insertions(+), 65 deletions(-) diff --git a/bins/tests/integration_test.rs b/bins/tests/integration_test.rs index d245748d..91d4d423 100644 --- a/bins/tests/integration_test.rs @@ -34,59 +34,79 @@ fn two_peers_and_a_seed() { let seed_home = "/tmp/seed-home"; let passphrase = b"play\n"; - println!("\n== create lnk homes for two peers and one seed ==\n"); - let cmd = LnkCmd::ProfileCreate; - run_lnk!(&cmd, peer1_home, passphrase); - run_lnk!(&cmd, peer2_home, passphrase); - run_lnk!(&cmd, seed_home, passphrase); + { + println!("\n== create lnk homes for two peers and one seed ==\n"); + let cmd = LnkCmd::ProfileCreate; + run_lnk!(&cmd, peer1_home, passphrase); + run_lnk!(&cmd, peer2_home, passphrase); + run_lnk!(&cmd, seed_home, passphrase); + } - println!("\n== add ssh keys for each profile to the ssh-agent ==\n"); - let cmd = LnkCmd::ProfileSshAdd; - run_lnk!(&cmd, peer1_home, passphrase); - run_lnk!(&cmd, peer2_home, passphrase); - run_lnk!(&cmd, seed_home, passphrase); - - println!("\n== Creating local link 1 identity ==\n"); - let peer1_name = "sockpuppet1".to_string(); - let cmd = LnkCmd::IdPersonCreate(peer1_name); - let output = run_lnk!(&cmd, peer1_home, passphrase); - let v: Value = serde_json::from_str(&output).unwrap(); - let urn1 = v["urn"].as_str().unwrap().to_string(); - let cmd = LnkCmd::IdLocalSet(urn1); - run_lnk!(&cmd, peer1_home, passphrase); - - println!("\n== Creating local link 2 identity ==\n"); - let peer2_name = "sockpuppet2".to_string(); - let cmd = LnkCmd::IdPersonCreate(peer2_name); - let output = run_lnk!(&cmd, peer2_home, passphrase); - let v: Value = serde_json::from_str(&output).unwrap(); - let urn2 = v["urn"].as_str().unwrap().to_string(); - let cmd = LnkCmd::IdLocalSet(urn2); - run_lnk!(&cmd, peer2_home, passphrase); + { + println!("\n== add ssh keys for each profile to the ssh-agent ==\n"); + let cmd = LnkCmd::ProfileSshAdd; + run_lnk!(&cmd, peer1_home, passphrase); + run_lnk!(&cmd, peer2_home, passphrase); + run_lnk!(&cmd, seed_home, passphrase); + } - println!("\n== Create a local repository ==\n"); - let peer1_proj = format!("peer1_proj_{}", timestamp()); - let cmd = LnkCmd::IdProjectCreate(peer1_proj.clone()); - let output = run_lnk!(&cmd, peer1_home, passphrase); - let v: Value = serde_json::from_str(&output).unwrap(); - let proj_urn = v["urn"].as_str().unwrap().to_string(); - println!("our project URN: {}", &proj_urn); + { + println!("\n== Creating local link 1 identity ==\n"); + let name = "sockpuppet1".to_string(); + let cmd = LnkCmd::IdPersonCreate { name }; + let output = run_lnk!(&cmd, peer1_home, passphrase); + let v: Value = serde_json::from_str(&output).unwrap(); + let urn = v["urn"].as_str().unwrap().to_string(); + let cmd = LnkCmd::IdLocalSet { urn: urn }; + run_lnk!(&cmd, peer1_home, passphrase); + } - println!("\n== Add the seed to the local peer seed configs ==\n"); - let cmd = LnkCmd::ProfilePeer; - let seed_peer_id = run_lnk!(&cmd, seed_home, passphrase); - let seed_endpoint = format!("{}@127.0.0.1:8799", &seed_peer_id); + { + println!("\n== Creating local link 2 identity ==\n"); + let name = "sockpuppet2".to_string(); + let cmd = LnkCmd::IdPersonCreate { name }; + let output = run_lnk!(&cmd, peer2_home, passphrase); + let v: Value = serde_json::from_str(&output).unwrap(); + let urn = v["urn"].as_str().unwrap().to_string(); + let cmd = LnkCmd::IdLocalSet { urn }; + run_lnk!(&cmd, peer2_home, passphrase); + } + + let urn = { + println!("\n== Create a local repository ==\n"); + let peer1_proj = format!("peer1_proj_{}", timestamp()); + let cmd = LnkCmd::IdProjectCreate { + name: peer1_proj.clone(), + }; + let output = run_lnk!(&cmd, peer1_home, passphrase); + let v: Value = serde_json::from_str(&output).unwrap(); + let proj_urn = v["urn"].as_str().unwrap().to_string(); + println!("our project URN: {}", &proj_urn); + proj_urn + }; - let cmd = LnkCmd::ProfileGet; - let peer1_profile = run_lnk!(&cmd, peer1_home, passphrase); - let peer1_seed = format!("{}/{}/seeds", peer1_home, peer1_profile); - let mut peer1_f = File::create(peer1_seed).unwrap(); - peer1_f.write_all(seed_endpoint.as_bytes()).unwrap(); + let seed_endpoint = { + println!("\n== Add the seed to the local peer seed configs ==\n"); + let cmd = LnkCmd::ProfilePeer; + let seed_peer_id = run_lnk!(&cmd, seed_home, passphrase); + format!("{}@127.0.0.1:8799", &seed_peer_id) + }; - let peer2_profile = run_lnk!(&cmd, peer2_home, passphrase); - let peer2_seed = format!("{}/{}/seeds", peer2_home, peer2_profile); - let mut peer2_f = File::create(peer2_seed).unwrap(); - peer2_f.write_all(seed_endpoint.as_bytes()).unwrap(); + { + let cmd = LnkCmd::ProfileGet; + let peer1_profile = run_lnk!(&cmd, peer1_home, passphrase); + let peer1_seed = format!("{}/{}/seeds", peer1_home, peer1_profile); + let mut peer1_f = File::create(peer1_seed).unwrap(); + peer1_f.write_all(seed_endpoint.as_bytes()).unwrap(); + } + + { + let cmd = LnkCmd::ProfileGet; + let peer2_profile = run_lnk!(&cmd, peer2_home, passphrase); + let peer2_seed = format!("{}/{}/seeds", peer2_home, peer2_profile); + let mut peer2_f = File::create(peer2_seed).unwrap(); + peer2_f.write_all(seed_endpoint.as_bytes()).unwrap(); + } println!("\n== Start the seed linkd ==\n"); let manifest_path = manifest_path(); @@ -99,7 +119,7 @@ fn two_peers_and_a_seed() { let mut gitd = spawn_lnk_gitd(peer1_home, &manifest_path, gitd_addr); println!("\n== Make some changes in the repo ==\n"); - env::set_current_dir(&peer1_proj).unwrap(); + env::set_current_dir(&urn).unwrap(); let mut test_file = File::create("test").unwrap(); test_file.write_all(b"test").unwrap(); Command::new("git") @@ -116,7 +136,7 @@ fn two_peers_and_a_seed() { println!("git-commit: {:?}", &output); println!("\n== Add the linkd remote to the repo ==\n"); - let remote_url = format!("ssh://rad@{}/{}.git", gitd_addr, &proj_urn); + let remote_url = format!("ssh://rad@{}/{}.git", gitd_addr, &urn); Command::new("git") .arg("remote") .arg("add") @@ -138,11 +158,12 @@ fn two_peers_and_a_seed() { env::set_current_dir("..").unwrap(); // out of the peer1 proj directory. let peer2_proj = format!("peer2_proj_{}", timestamp()); - let cmd = LnkCmd::Clone(proj_urn, peer1_peer_id, peer2_proj.clone()); + let cmd = LnkCmd::Clone { + urn, + peer_id: peer1_peer_id, + path: peer2_proj.clone(), + }; run_lnk!(&cmd, peer2_home, passphrase); - if !is_parent { - return; - } env::set_current_dir(peer2_proj).unwrap(); let peer2_last_commit = git_last_commit(); @@ -162,10 +183,20 @@ enum LnkCmd { ProfileGet, ProfilePeer, ProfileSshAdd, - IdPersonCreate(String), // the associated string is "the person's name". - IdLocalSet(String), // the associated string is "urn". - IdProjectCreate(String), // the associated string is "the project name". - Clone(String, String, String), // the associated string is "urn", "peer_id", "path" + IdPersonCreate { + name: String, + }, + IdLocalSet { + urn: String, + }, + IdProjectCreate { + name: String, + }, + Clone { + urn: String, + peer_id: String, + path: String, + }, } /// Runs a `lnk` command of `$cmd` using `$lnk_home` as the node home. @@ -199,12 +230,12 @@ fn process_lnk_output(lnk_home: &str, lnk_process: &mut Master, cmd: &LnkCmd) -> // Print the output and decode them if necessary. println!("{}: {}", lnk_home, line); match cmd { - LnkCmd::IdPersonCreate(ref _name) => { + LnkCmd::IdPersonCreate { .. } => { if line.find("\"urn\":").is_some() { output = line; // get the line with URN. } }, - LnkCmd::IdProjectCreate(ref _name) => { + LnkCmd::IdProjectCreate { .. } => { if line.find("\"urn\":").is_some() { output = line; // get the line with URN. } @@ -215,7 +246,7 @@ fn process_lnk_output(lnk_home: &str, lnk_process: &mut Master, cmd: &LnkCmd) -> LnkCmd::ProfilePeer => { output = line; // get the last line for peer id. }, - LnkCmd::Clone(ref _urn, ref _peer, ref _path) => { + LnkCmd::Clone { .. } => { output = line; }, _ => {}, @@ -245,7 +276,7 @@ fn start_lnk_cmd(lnk_home: &str, cmd: &LnkCmd) { LnkCmd::ProfileGet => lnk_cmd.arg("profile").arg("get"), LnkCmd::ProfilePeer => lnk_cmd.arg("profile").arg("peer"), LnkCmd::ProfileSshAdd => lnk_cmd.arg("profile").arg("ssh").arg("add"), - LnkCmd::IdPersonCreate(name) => { + LnkCmd::IdPersonCreate { name } => { let payload = json!({ "name": name }); lnk_cmd .arg("identities") @@ -255,13 +286,13 @@ fn start_lnk_cmd(lnk_home: &str, cmd: &LnkCmd) { .arg("--payload") .arg(payload.to_string()) }, - LnkCmd::IdLocalSet(urn) => lnk_cmd + LnkCmd::IdLocalSet { urn } => lnk_cmd .arg("identities") .arg("local") .arg("set") .arg("--urn") .arg(urn), - LnkCmd::IdProjectCreate(name) => { + LnkCmd::IdProjectCreate { name } => { let payload = json!({"name": name, "default_branch": "master"}); let project_path = format!("./{}", name); lnk_cmd @@ -274,7 +305,11 @@ fn start_lnk_cmd(lnk_home: &str, cmd: &LnkCmd) { .arg("--payload") .arg(payload.to_string()) }, - LnkCmd::Clone(urn, peer_id, peer2_proj) => lnk_cmd + LnkCmd::Clone { + urn, + peer_id, + path: peer2_proj, + } => lnk_cmd .arg("clone") .arg("--urn") .arg(urn) -- 2.31.1
On Mon, Aug 29, 2022 at 12:13 PM Fintan Halpenny <fintan.halpenny@gmail.com> wrote:
--- bins/Cargo.lock | 29 +++ bins/Cargo.toml | 1 + bins/tests/Cargo.toml | 13 ++ bins/tests/integration_test.rs | 392 +++++++++++++++++++++++++++++++++ 4 files changed, 435 insertions(+) create mode 100644 bins/tests/Cargo.toml create mode 100644 bins/tests/integration_test.rs diff --git a/bins/Cargo.lock b/bins/Cargo.lock index d5756144..4d0ea413 100644 --- a/bins/Cargo.lock @@ -838,6 +838,17 @@ dependencies = [ "termcolor", ] +[[package]] +name = "errno" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e2b2decb0484e15560df3210cf0d78654bb0864b2c138977c07e377a1bae0e2" +dependencies = [ + "kernel32-sys", + "libc", + "winapi 0.2.8", +] + [[package]] name = "event-listener" version = "2.5.2" @@ -2945,6 +2956,16 @@ dependencies = [ "human_format", ] +[[package]] +name = "pty" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f50f3d255966981eb4e4c5df3e983e6f7d163221f547406d83b6a460ff5c5ee8" +dependencies = [ + "errno", + "libc", +] + [[package]] name = "quanta" version = "0.4.1" @@ -3567,6 +3588,14 @@ dependencies = [ "winapi-util", ] +[[package]] +name = "tests" +version = "0.1.0" +dependencies = [ + "pty", + "serde_json", +] + [[package]] name = "textwrap" version = "0.15.0" diff --git a/bins/Cargo.toml b/bins/Cargo.toml index 3d1786f4..99b3c69f 100644 --- a/bins/Cargo.toml @@ -5,4 +5,5 @@ members = [ "lnk-gitd", "lnk-identities-dev", "lnk-profile-dev", + "tests", ] diff --git a/bins/tests/Cargo.toml b/bins/tests/Cargo.toml new file mode 100644 index 00000000..3ae82769 --- /dev/null @@ -0,0 +1,13 @@ +[package] +name = "tests" +version = "0.1.0" +authors = ["Han Xu <keepsimple@gmail.com>"] +edition = "2021" + +[dev-dependencies] +pty = "0.2" +serde_json = "1.0" + +[[test]] +name = "integration_test" +path = "integration_test.rs" diff --git a/bins/tests/integration_test.rs b/bins/tests/integration_test.rs new file mode 100644 index 00000000..4ff581a0 --- /dev/null @@ -0,0 +1,392 @@ +// Copyright © 2022 The Radicle Link Contributors +// +// This file is part of radicle-link, distributed under the GPLv3 with Radicle +// Linking Exception. For full terms see the included LICENSE file. + +//! Integration test to exercise the programs in `bins`. + +use pty::fork::*; +use serde_json::{json, Value}; +use std::{ + env, + fs::File, + io::{BufRead, BufReader, Write}, + process::{Child, Command, Stdio}, + thread, + time::{Duration, SystemTime}, +}; + +/// This test is inspired by https://github.com/alexjg/linkd-playground +#[test] +fn happy_path_to_push_changes() { + let peer1_home = "/tmp/link-local-1"; + let peer2_home = "/tmp/link-local-2"; + let seed_home = "/tmp/seed-home"; + let passphrase = b"play\n"; + + println!("\n== create lnk homes for two peers and one seed ==\n"); + let (is_parent, _) = run_lnk(LnkCmd::ProfileCreate, peer1_home, passphrase); + if !is_parent { + return; + } + let (is_parent, _) = run_lnk(LnkCmd::ProfileCreate, peer2_home, passphrase); + if !is_parent { + return; + } + let (is_parent, _) = run_lnk(LnkCmd::ProfileCreate, seed_home, passphrase); + if !is_parent { + return; + } + + println!("\n== add ssh keys for each profile to the ssh-agent ==\n"); + let (is_parent, _) = run_lnk(LnkCmd::ProfileSshAdd, peer1_home, passphrase); + if !is_parent { + return; + } + let (is_parent, _) = run_lnk(LnkCmd::ProfileSshAdd, peer2_home, passphrase); + if !is_parent { + return; + } + let (is_parent, _) = run_lnk(LnkCmd::ProfileSshAdd, seed_home, passphrase); + if !is_parent { + return; + } + + println!("\n== Creating local link 1 identity ==\n"); + let peer1_name = "sockpuppet1".to_string(); + let (is_parent, output) = run_lnk(LnkCmd::IdPersonCreate(peer1_name), peer1_home, passphrase); + if !is_parent { + return; + } + let v: Value = serde_json::from_str(&output).unwrap(); + let urn1 = v["urn"].as_str().unwrap().to_string(); + let (is_parent, _) = run_lnk(LnkCmd::IdLocalSet(urn1), peer1_home, passphrase); + if !is_parent { + return; + } + + println!("\n== Creating local link 2 identity ==\n"); + let peer2_name = "sockpuppet2".to_string(); + let (is_parent, output) = run_lnk(LnkCmd::IdPersonCreate(peer2_name), peer2_home, passphrase); + if !is_parent { + return; + } + let v: Value = serde_json::from_str(&output).unwrap(); + let urn2 = v["urn"].as_str().unwrap().to_string(); + let (is_parent, _) = run_lnk(LnkCmd::IdLocalSet(urn2), peer2_home, passphrase); + if !is_parent { + return; + } + + println!("\n== Create a local repository ==\n"); + let peer1_proj = format!("peer1_proj_{}", timestamp()); + let (is_parent, output) = run_lnk( + LnkCmd::IdProjectCreate(peer1_proj.clone()), + peer1_home, + passphrase, + ); + if !is_parent { + return; + } + let v: Value = serde_json::from_str(&output).unwrap(); + let proj_urn = v["urn"].as_str().unwrap().to_string(); + println!("our project URN: {}", &proj_urn); + + println!("\n== Add the seed to the local peer seed configs ==\n"); + let (is_parent, seed_peer_id) = run_lnk(LnkCmd::ProfilePeer, seed_home, passphrase); + if !is_parent { + return; + } + let seed_endpoint = format!("{}@127.0.0.1:8799", &seed_peer_id); + + let (is_parent, peer1_profile) = run_lnk(LnkCmd::ProfileGet, peer1_home, passphrase); + if !is_parent { + return; + } + let peer1_seed = format!("{}/{}/seeds", peer1_home, peer1_profile); + let mut peer1_f = File::create(peer1_seed).unwrap(); + peer1_f.write_all(seed_endpoint.as_bytes()).unwrap(); + + let (is_parent, peer2_profile) = run_lnk(LnkCmd::ProfileGet, peer2_home, passphrase); + if !is_parent { + return; + } + let peer2_seed = format!("{}/{}/seeds", peer2_home, peer2_profile); + let mut peer2_f = File::create(peer2_seed).unwrap(); + peer2_f.write_all(seed_endpoint.as_bytes()).unwrap(); + + println!("\n== Start the seed linkd ==\n"); + let manifest_path = manifest_path(); + let mut linkd = spawn_linkd(seed_home, &manifest_path); + + println!("\n== Start the peer 1 gitd ==\n"); + let (is_parent, peer1_peer_id) = run_lnk(LnkCmd::ProfilePeer, peer1_home, passphrase); + if !is_parent { + return; + } + let mut lnk_gitd = spawn_lnk_gitd(peer1_home, &manifest_path, &peer1_peer_id); + + println!("\n== Make some changes in the repo ==\n"); + env::set_current_dir(&peer1_proj).unwrap(); + let mut test_file = File::create("test").unwrap(); + test_file.write_all(b"test").unwrap(); + Command::new("git") + .arg("add") + .arg("test") + .output() + .expect("failed to do git add"); + let output = Command::new("git") + .arg("commit") + .arg("-m") + .arg("test commit") + .output() + .expect("failed to do git commit"); + println!("git-commit: {:?}", &output); + + println!("\n== Add the linkd remote to the repo ==\n"); + let remote_url = format!("ssh://rad@127.0.0.1:9987/{}.git", &proj_urn); + Command::new("git") + .arg("remote") + .arg("add") + .arg("linkd") + .arg(remote_url) + .output() + .expect("failed to do git remote add"); + + clean_up_known_hosts(); + + linkd.kill().ok(); + lnk_gitd.kill().ok(); +} + +enum LnkCmd { + ProfileCreate, + ProfileGet, + ProfilePeer, + ProfileSshAdd, + IdPersonCreate(String), // the associated string is "the person's name". + IdLocalSet(String), // the associated string is "urn". + IdProjectCreate(String), // the associated string is "the project name". +} + +/// Runs a `cmd` for `lnk_home`. Rebuilds `lnk` if necessary. +/// Return.0: true if this is the parent (i.e. test) process, +/// false if this is the child (i.e. lnk) process. +/// Return.1: an output that depends on the `cmd`. +fn run_lnk(cmd: LnkCmd, lnk_home: &str, passphrase: &[u8]) -> (bool, String) { + let fork = Fork::from_ptmx().unwrap(); + if let Some(mut parent) = fork.is_parent().ok() { + // Input the passphrase if necessary. + match cmd { + LnkCmd::ProfileCreate | LnkCmd::ProfileSshAdd => { + parent.write_all(passphrase).unwrap(); + println!("{}: wrote passphase", lnk_home); + }, + _ => {}, + } + + // Print the output and decode them if necessary. + let buf_reader = BufReader::new(parent); + let mut output = String::new(); + for line in buf_reader.lines() { + let line = line.unwrap(); + println!("{}: {}", lnk_home, line); + + match cmd { + LnkCmd::IdPersonCreate(ref _name) => { + if line.find("\"urn\":").is_some() { + output = line; // get the line with URN. + } + }, + LnkCmd::IdProjectCreate(ref _name) => { + if line.find("\"urn\":").is_some() { + output = line; // get the line with URN. + } + }, + LnkCmd::ProfileGet => { + output = line; // get the last line for profile id. + }, + LnkCmd::ProfilePeer => { + output = line; // get the last line for peer id. + }, + _ => {}, + } + } + + (true, output) + } else { + // Child process is to run `lnk`. + let manifest_path = manifest_path(); + + // cargo run \ + // --manifest-path $LINK_CHECKOUT/bins/Cargo.toml \ + // -p lnk -- "$@" + let mut lnk_cmd = Command::new("cargo"); + lnk_cmd + .env("LNK_HOME", lnk_home) + .arg("run") + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("lnk") + .arg("--"); + let full_cmd = match cmd { + LnkCmd::ProfileCreate => lnk_cmd.arg("profile").arg("create"), + LnkCmd::ProfileGet => lnk_cmd.arg("profile").arg("get"), + LnkCmd::ProfilePeer => lnk_cmd.arg("profile").arg("peer"), + LnkCmd::ProfileSshAdd => lnk_cmd.arg("profile").arg("ssh").arg("add"), + LnkCmd::IdPersonCreate(name) => { + let payload = json!({ "name": name }); + lnk_cmd + .arg("identities") + .arg("person") + .arg("create") + .arg("new") + .arg("--payload") + .arg(payload.to_string()) + }, + LnkCmd::IdLocalSet(urn) => lnk_cmd + .arg("identities") + .arg("local") + .arg("set") + .arg("--urn") + .arg(urn), + LnkCmd::IdProjectCreate(name) => { + let payload = json!({"name": name, "default_branch": "master"}); + let project_path = format!("./{}", name); + lnk_cmd + .arg("identities") + .arg("project") + .arg("create") + .arg("new") + .arg("--path") + .arg(project_path) + .arg("--payload") + .arg(payload.to_string()) + }, + }; + full_cmd.status().expect("lnk cmd failed:"); + + (false, String::new()) + } +} + +fn spawn_linkd(lnk_home: &str, manifest_path: &str) -> Child { + let log_name = format!("linkd_{}.log", ×tamp()); + let log_file = File::create(&log_name).unwrap(); + let child = Command::new("cargo") + .arg("run") + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("linkd") + .arg("--") + .arg("--lnk-home") + .arg(lnk_home) + .arg("--track") + .arg("everything") + .arg("--protocol-listen") + .arg("127.0.0.1:8799") + .stdout(Stdio::from(log_file)) + .spawn() + .expect("linkd failed to start"); + println!("linkd stdout redirected to {}", &log_name); + thread::sleep(Duration::from_secs(1)); + child +} + +fn spawn_lnk_gitd(lnk_home: &str, manifest_path: &str, peer_id: &str) -> Child { + let log_name = format!("lnk-gitd_{}.log", ×tamp()); + let log_file = File::create(&log_name).unwrap(); + let port = "9987"; + let xdg_runtime_dir = env!("XDG_RUNTIME_DIR"); + let rpc_socket = format!("{}/link-peer-{}-rpc.socket", xdg_runtime_dir, peer_id); + + Command::new("cargo") + .arg("build") + .arg("--target-dir") + .arg("./target") + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("lnk-gitd") + .output() + .expect("cargo build lnk-gitd failed"); + + let child = Command::new("systemd-socket-activate") + .arg("-l") + .arg(port) + .arg("--fdname=ssh") + .arg("-E") + .arg("SSH_AUTH_SOCK") + .arg("-E") + .arg("RUST_BACKTRACE") + .arg("./target/debug/lnk-gitd") + .arg(lnk_home) + .arg("--linkd-rpc-socket") + .arg(rpc_socket) + .arg("--push-seeds") + .arg("--fetch-seeds") + .arg("--linger-timeout") + .arg("10000") + .stdout(Stdio::from(log_file)) + .spawn() + .expect("lnk-gitd failed to start"); + println!("lnk-gitd stdout redirected to {}", &log_name); + thread::sleep(Duration::from_secs(1)); + + child +} + +/// Returns true if this is the parent process, +/// returns false if this is the child process. +fn _run_git_push() -> bool { + let fork = Fork::from_ptmx().unwrap(); + if let Some(mut parent) = fork.is_parent().ok() { + let yes = b"yes\n"; + parent.write_all(yes).unwrap(); + + let buf_reader = BufReader::new(parent); + for line in buf_reader.lines() { + let line = line.unwrap(); + println!("git-push: {}", line); + } + + true + } else { + Command::new("git") + .arg("push") + .arg("linkd") + .status() + .expect("failed to do git push"); + false + } +} + +/// Returns UNIX_TIME in millis. +fn timestamp() -> u128 { + let now = SystemTime::now() + .duration_since(SystemTime::UNIX_EPOCH) + .unwrap(); + now.as_millis() +} + +/// Returns the full path of `bins` manifest file. +fn manifest_path() -> String { + let package_dir = env!("CARGO_MANIFEST_DIR"); + format!("{}/Cargo.toml", package_dir.strip_suffix("/tests").unwrap()) +} + +fn clean_up_known_hosts() { + // ssh-keygen -f "/home/pi/.ssh/known_hosts" -R "[127.0.0.1]:9987" + let home_dir = env!("HOME"); + let known_hosts = format!("{}/.ssh/known_hosts", &home_dir); + let output = Command::new("ssh-keygen") + .arg("-f") + .arg(known_hosts) + .arg("-R") + .arg("[127.0.0.1]:9987") + .output() + .expect("failed to do ssh-keygen"); + println!("ssh-keygen: {:?}", &output); +} -- 2.32.0 (Apple Git-132)
builds.sr.ht <builds@sr.ht>radicle-link/patches/nixos-latest.yml: FAILED in 23m57s [add integration test for bins package][0] from [Han Xu][1] [0]: https://lists.sr.ht/~radicle-link/dev/patches/34576 [1]: mailto:keepsimple@gmail.com ✗ #820143 FAILED radicle-link/patches/nixos-latest.yml https://builds.sr.ht/~radicle-link/job/820143
--- bins/Cargo.lock | 29 +++ bins/Cargo.toml | 1 + bins/tests/Cargo.toml | 13 + bins/tests/integration_test.rs | 458 +++++++++++++++++++++++++++++++++ link-git/src/protocol/fetch.rs | 2 +- link-git/src/protocol/ls.rs | 2 +- 6 files changed, 503 insertions(+), 2 deletions(-) create mode 100644 bins/tests/Cargo.toml create mode 100644 bins/tests/integration_test.rs diff --git a/bins/Cargo.lock b/bins/Cargo.lock index d5756144..4d0ea413 100644 --- a/bins/Cargo.lock @@ -838,6 +838,17 @@ dependencies = [ "termcolor", ] +[[package]] +name = "errno" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e2b2decb0484e15560df3210cf0d78654bb0864b2c138977c07e377a1bae0e2" +dependencies = [ + "kernel32-sys", + "libc", + "winapi 0.2.8", +] + [[package]] name = "event-listener" version = "2.5.2" @@ -2945,6 +2956,16 @@ dependencies = [ "human_format", ] +[[package]] +name = "pty" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f50f3d255966981eb4e4c5df3e983e6f7d163221f547406d83b6a460ff5c5ee8" +dependencies = [ + "errno", + "libc", +] + [[package]] name = "quanta" version = "0.4.1" @@ -3567,6 +3588,14 @@ dependencies = [ "winapi-util", ] +[[package]] +name = "tests" +version = "0.1.0" +dependencies = [ + "pty", + "serde_json", +] + [[package]] name = "textwrap" version = "0.15.0" diff --git a/bins/Cargo.toml b/bins/Cargo.toml index 3d1786f4..99b3c69f 100644 --- a/bins/Cargo.toml @@ -5,4 +5,5 @@ members = [ "lnk-gitd", "lnk-identities-dev", "lnk-profile-dev", + "tests", ] diff --git a/bins/tests/Cargo.toml b/bins/tests/Cargo.toml new file mode 100644 index 00000000..3ae82769 --- /dev/null @@ -0,0 +1,13 @@ +[package] +name = "tests" +version = "0.1.0" +authors = ["Han Xu <keepsimple@gmail.com>"] +edition = "2021" + +[dev-dependencies] +pty = "0.2" +serde_json = "1.0" + +[[test]] +name = "integration_test" +path = "integration_test.rs" diff --git a/bins/tests/integration_test.rs b/bins/tests/integration_test.rs new file mode 100644 index 00000000..d6d3b4ae --- /dev/null @@ -0,0 +1,458 @@ +// Copyright © 2022 The Radicle Link Contributors +// +// This file is part of radicle-link, distributed under the GPLv3 with Radicle +// Linking Exception. For full terms see the included LICENSE file. + +//! Integration test to exercise the programs in `bins`. + +use pty::fork::*; +use serde_json::{json, Value}; +use std::{ + env, + fs::File, + io::{BufRead, BufReader, Write}, + process::{Child, Command, Stdio}, + thread, + time::{Duration, SystemTime}, +}; + +/// This test is inspired by https://github.com/alexjg/linkd-playground +/// +/// Tests a typical scenario: there are two peer nodes and one seed node. +/// The main steps are: +/// - Setup a profile for each node in tmp home directories. +/// - Setup SSH keys for each profile. +/// - Create identities. +/// - Create a local repo for peer 1. +/// - Start `linkd` for the seed, and `lnk-gitd` for peer 1. +/// - Push peer 1 repo to its monorepo and to the seed. +/// - Clone the peer 1 repo to peer 2 via seed. +#[test] +fn two_peers_and_a_seed() { + let peer1_home = "/tmp/link-local-1"; + let peer2_home = "/tmp/link-local-2"; + let seed_home = "/tmp/seed-home"; + let passphrase = b"play\n"; + + println!("\n== create lnk homes for two peers and one seed ==\n"); + let (is_parent, _) = run_lnk(LnkCmd::ProfileCreate, peer1_home, passphrase); + if !is_parent { + return; + } + let (is_parent, _) = run_lnk(LnkCmd::ProfileCreate, peer2_home, passphrase); + if !is_parent { + return; + } + let (is_parent, _) = run_lnk(LnkCmd::ProfileCreate, seed_home, passphrase); + if !is_parent { + return; + } + + println!("\n== add ssh keys for each profile to the ssh-agent ==\n"); + let (is_parent, _) = run_lnk(LnkCmd::ProfileSshAdd, peer1_home, passphrase); + if !is_parent { + return; + } + let (is_parent, _) = run_lnk(LnkCmd::ProfileSshAdd, peer2_home, passphrase); + if !is_parent { + return; + } + let (is_parent, _) = run_lnk(LnkCmd::ProfileSshAdd, seed_home, passphrase); + if !is_parent { + return; + } + + println!("\n== Creating local link 1 identity ==\n"); + let peer1_name = "sockpuppet1".to_string(); + let (is_parent, output) = run_lnk(LnkCmd::IdPersonCreate(peer1_name), peer1_home, passphrase); + if !is_parent { + return; + } + let v: Value = serde_json::from_str(&output).unwrap(); + let urn1 = v["urn"].as_str().unwrap().to_string(); + let (is_parent, _) = run_lnk(LnkCmd::IdLocalSet(urn1), peer1_home, passphrase); + if !is_parent { + return; + } + + println!("\n== Creating local link 2 identity ==\n"); + let peer2_name = "sockpuppet2".to_string(); + let (is_parent, output) = run_lnk(LnkCmd::IdPersonCreate(peer2_name), peer2_home, passphrase); + if !is_parent { + return; + } + let v: Value = serde_json::from_str(&output).unwrap(); + let urn2 = v["urn"].as_str().unwrap().to_string(); + let (is_parent, _) = run_lnk(LnkCmd::IdLocalSet(urn2), peer2_home, passphrase); + if !is_parent { + return; + } + + println!("\n== Create a local repository ==\n"); + let peer1_proj = format!("peer1_proj_{}", timestamp()); + let (is_parent, output) = run_lnk( + LnkCmd::IdProjectCreate(peer1_proj.clone()), + peer1_home, + passphrase, + ); + if !is_parent { + return; + } + let v: Value = serde_json::from_str(&output).unwrap(); + let proj_urn = v["urn"].as_str().unwrap().to_string(); + println!("our project URN: {}", &proj_urn); + + println!("\n== Add the seed to the local peer seed configs ==\n"); + let (is_parent, seed_peer_id) = run_lnk(LnkCmd::ProfilePeer, seed_home, passphrase); + if !is_parent { + return; + } + let seed_endpoint = format!("{}@127.0.0.1:8799", &seed_peer_id); + + let (is_parent, peer1_profile) = run_lnk(LnkCmd::ProfileGet, peer1_home, passphrase); + if !is_parent { + return; + } + let peer1_seed = format!("{}/{}/seeds", peer1_home, peer1_profile); + let mut peer1_f = File::create(peer1_seed).unwrap(); + peer1_f.write_all(seed_endpoint.as_bytes()).unwrap(); + + let (is_parent, peer2_profile) = run_lnk(LnkCmd::ProfileGet, peer2_home, passphrase); + if !is_parent { + return; + } + let peer2_seed = format!("{}/{}/seeds", peer2_home, peer2_profile); + let mut peer2_f = File::create(peer2_seed).unwrap(); + peer2_f.write_all(seed_endpoint.as_bytes()).unwrap(); + + println!("\n== Start the seed linkd ==\n"); + let manifest_path = manifest_path(); + let mut linkd = spawn_linkd(seed_home, &manifest_path); + + println!("\n== Start the peer 1 gitd ==\n"); + let (is_parent, peer1_peer_id) = run_lnk(LnkCmd::ProfilePeer, peer1_home, passphrase); + if !is_parent { + return; + } + let is_parent = spawn_lnk_gitd(peer1_home, &manifest_path, &peer1_peer_id); + if !is_parent { + return; + } + + println!("\n== Make some changes in the repo ==\n"); + env::set_current_dir(&peer1_proj).unwrap(); + let mut test_file = File::create("test").unwrap(); + test_file.write_all(b"test").unwrap(); + Command::new("git") + .arg("add") + .arg("test") + .output() + .expect("failed to do git add"); + let output = Command::new("git") + .arg("commit") + .arg("-m") + .arg("test commit") + .output() + .expect("failed to do git commit"); + println!("git-commit: {:?}", &output); + + println!("\n== Add the linkd remote to the repo ==\n"); + let remote_url = format!("ssh://rad@127.0.0.1:9987/{}.git", &proj_urn); + Command::new("git") + .arg("remote") + .arg("add") + .arg("linkd") + .arg(remote_url) + .output() + .expect("failed to do git remote add"); + + clean_up_known_hosts(); + + let is_parent = run_git_push(); + if !is_parent { + return; + } + + println!("\n== Clone to peer2 ==\n"); + + env::set_current_dir("..").unwrap(); // out of the peer1 proj directory. + let (is_parent, _) = run_lnk( + LnkCmd::Clone(proj_urn, peer1_peer_id), + peer2_home, + passphrase, + ); + if !is_parent { + return; + } + + println!("\n== Kill linkd (seed) ==\n"); + + linkd.kill().ok(); +} + +enum LnkCmd { + ProfileCreate, + ProfileGet, + ProfilePeer, + ProfileSshAdd, + IdPersonCreate(String), // the associated string is "the person's name". + IdLocalSet(String), // the associated string is "urn". + IdProjectCreate(String), // the associated string is "the project name". + Clone(String, String), // the associated string is "urn", "peer_id" +} + +/// Runs a `cmd` for `lnk_home`. Rebuilds `lnk` if necessary. +/// Return.0: true if this is the parent (i.e. test) process, +/// false if this is the child (i.e. lnk) process. +/// Return.1: an output that depends on the `cmd`. +fn run_lnk(cmd: LnkCmd, lnk_home: &str, passphrase: &[u8]) -> (bool, String) { + let fork = Fork::from_ptmx().unwrap(); + if let Some(mut parent) = fork.is_parent().ok() { + // Input the passphrase if necessary. + match cmd {
Alex Good <alex@memoryandthought.me>What's the reasoning behing using this API where we fork and check if we're in a child vs using the std::process::Command APIs?
+ LnkCmd::ProfileCreate | LnkCmd::ProfileSshAdd => { + parent.write_all(passphrase).unwrap(); + println!("{}: wrote passphase", lnk_home); + }, + _ => {}, + } + + // Print the output and decode them if necessary. + let buf_reader = BufReader::new(parent); + let mut output = String::new(); + for line in buf_reader.lines() { + let line = line.unwrap(); + println!("{}: {}", lnk_home, line); + + match cmd { + LnkCmd::IdPersonCreate(ref _name) => { + if line.find("\"urn\":").is_some() { + output = line; // get the line with URN. + } + }, + LnkCmd::IdProjectCreate(ref _name) => { + if line.find("\"urn\":").is_some() { + output = line; // get the line with URN. + } + }, + LnkCmd::ProfileGet => { + output = line; // get the last line for profile id. + }, + LnkCmd::ProfilePeer => { + output = line; // get the last line for peer id. + }, + LnkCmd::Clone(ref _urn, ref _peer) => { + output = line; + }, + _ => {}, + } + } + + (true, output) + } else { + // Child process is to run `lnk`. + let manifest_path = manifest_path(); + + // cargo run \ + // --manifest-path $LINK_CHECKOUT/bins/Cargo.toml \ + // -p lnk -- "$@" + let mut lnk_cmd = Command::new("cargo"); + lnk_cmd + .env("LNK_HOME", lnk_home) + .arg("run") + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("lnk") + .arg("--"); + let full_cmd = match cmd { + LnkCmd::ProfileCreate => lnk_cmd.arg("profile").arg("create"), + LnkCmd::ProfileGet => lnk_cmd.arg("profile").arg("get"), + LnkCmd::ProfilePeer => lnk_cmd.arg("profile").arg("peer"), + LnkCmd::ProfileSshAdd => lnk_cmd.arg("profile").arg("ssh").arg("add"), + LnkCmd::IdPersonCreate(name) => { + let payload = json!({ "name": name }); + lnk_cmd + .arg("identities") + .arg("person") + .arg("create") + .arg("new") + .arg("--payload") + .arg(payload.to_string()) + }, + LnkCmd::IdLocalSet(urn) => lnk_cmd + .arg("identities") + .arg("local") + .arg("set") + .arg("--urn") + .arg(urn), + LnkCmd::IdProjectCreate(name) => { + let payload = json!({"name": name, "default_branch": "master"}); + let project_path = format!("./{}", name); + lnk_cmd + .arg("identities") + .arg("project") + .arg("create") + .arg("new") + .arg("--path") + .arg(project_path) + .arg("--payload") + .arg(payload.to_string()) + }, + LnkCmd::Clone(urn, peer_id) => { + let peer2_proj = format!("peer2_proj_{}", timestamp()); + lnk_cmd + .arg("clone") + .arg("--urn") + .arg(urn) + .arg("--path") + .arg(peer2_proj) + .arg("--peer") + .arg(peer_id) + }, + }; + full_cmd.status().expect("lnk cmd failed:"); + + (false, String::new()) + } +} + +fn spawn_linkd(lnk_home: &str, manifest_path: &str) -> Child { + let log_name = format!("linkd_{}.log", ×tamp()); + let log_file = File::create(&log_name).unwrap(); + let target_dir = bins_target_dir(); + let exec_path = format!("{}/debug/linkd", &target_dir); + + Command::new("cargo") + .arg("build") + .arg("--target-dir") + .arg(&target_dir) + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("linkd") + .output() + .expect("cargo build linkd failed"); + + let child = Command::new(&exec_path) + .env("RUST_BACKTRACE", "1") + .arg("--lnk-home") + .arg(lnk_home) + .arg("--track") + .arg("everything") + .arg("--protocol-listen") + .arg("127.0.0.1:8799") + .stdout(Stdio::from(log_file)) + .spawn() + .expect("linkd failed to start"); + println!("linkd stdout redirected to {}", &log_name); + thread::sleep(Duration::from_secs(1)); + child +} + +fn spawn_lnk_gitd(lnk_home: &str, manifest_path: &str, peer_id: &str) -> bool { + let port = "9987"; + let xdg_runtime_dir = env!("XDG_RUNTIME_DIR"); + let rpc_socket = format!("{}/link-peer-{}-rpc.socket", xdg_runtime_dir, peer_id); + let target_dir = bins_target_dir(); + let exec_path = format!("{}/debug/lnk-gitd", &target_dir); + + Command::new("cargo") + .arg("build") + .arg("--target-dir") + .arg(&target_dir) + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("lnk-gitd") + .stdout(Stdio::inherit()) + .output() + .expect("cargo build lnk-gitd failed"); + + let fork = Fork::from_ptmx().unwrap(); + + if let Some(_parent) = fork.is_parent().ok() { + println!("started lnk-gitd"); + true + } else { + Command::new("systemd-socket-activate") + .arg("-l") + .arg(port) + .arg("--fdname=ssh") + .arg("-E") + .arg("SSH_AUTH_SOCK") + .arg("-E") + .arg("RUST_BACKTRACE") + .arg(&exec_path) + .arg(lnk_home) + .arg("--linkd-rpc-socket") + .arg(rpc_socket) + .arg("--push-seeds") + .arg("--fetch-seeds") + .arg("--linger-timeout") + .arg("10000") + .output() + .expect("lnk-gitd failed to start"); + false + } +} + +/// Returns true if this is the parent process, +/// returns false if this is the child process. +fn run_git_push() -> bool { + let fork = Fork::from_ptmx().unwrap(); + if let Some(mut parent) = fork.is_parent().ok() { + let yes = b"yes\n"; + let buf_reader = BufReader::new(parent); + for line in buf_reader.lines() { + let line = line.unwrap(); + println!("git-push: {}", line); + if line.find("key fingerprint").is_some() { + parent.write_all(yes).unwrap(); + } + } + + true + } else { + Command::new("git") + .arg("push") + .arg("linkd") + .status() + .expect("failed to do git push"); + false + } +} + +/// Returns UNIX_TIME in millis. +fn timestamp() -> u128 { + let now = SystemTime::now() + .duration_since(SystemTime::UNIX_EPOCH) + .unwrap(); + now.as_millis() +} + +/// Returns the full path of `bins` manifest file. +fn manifest_path() -> String { + let package_dir = env!("CARGO_MANIFEST_DIR"); + format!("{}/Cargo.toml", package_dir.strip_suffix("/tests").unwrap()) +} + +/// Returns the full path of `bins/target`. +fn bins_target_dir() -> String { + let package_dir = env!("CARGO_MANIFEST_DIR"); + format!("{}/target", package_dir.strip_suffix("/tests").unwrap()) +} + +fn clean_up_known_hosts() { + // ssh-keygen -f "/home/pi/.ssh/known_hosts" -R "[127.0.0.1]:9987" + let home_dir = env!("HOME"); + let known_hosts = format!("{}/.ssh/known_hosts", &home_dir); + let output = Command::new("ssh-keygen") + .arg("-f") + .arg(known_hosts) + .arg("-R") + .arg("[127.0.0.1]:9987") + .output() + .expect("failed to do ssh-keygen"); + println!("ssh-keygen: {:?}", &output); +} diff --git a/link-git/src/protocol/fetch.rs b/link-git/src/protocol/fetch.rs index a5a73065..562e433d 100644 --- a/link-git/src/protocol/fetch.rs +++ b/link-git/src/protocol/fetch.rs @@ -39,7 +39,7 @@ use super::{packwriter::PackWriter, remote_git_version, transport}; // cf. https://lore.kernel.org/git/CD2XNXHACAXS.13J6JTWZPO1JA@schmidt/ // Fixed in `git.git` 1ab13eb, which should land in 2.34 fn must_namespace_want_ref(caps: &client::Capabilities) -> bool { - static FIXED_AFTER: Lazy<Version> = Lazy::new(|| Version::new("2.33.0").unwrap()); + static FIXED_AFTER: Lazy<Version> = Lazy::new(|| Version::new("2.25.0").unwrap()); remote_git_version(caps) .map(|version| version <= *FIXED_AFTER) diff --git a/link-git/src/protocol/ls.rs b/link-git/src/protocol/ls.rs
Alex Good <alex@memoryandthought.me>This is quite surprising to me but as long as we've tested it and it works that's great! We should probably change the comment above about it landing in 2.34 though.
index b3516454..9dafa538 100644 --- a/link-git/src/protocol/ls.rs +++ b/link-git/src/protocol/ls.rs @@ -24,7 +24,7 @@ use super::{remote_git_version, transport}; // cf. https://lore.kernel.org/git/pMV5dJabxOBTD8kJBaPuWK0aS6OJhRQ7YFGwfhPCeSJEbPDrIFBza36nXBCgUCeUJWGmpjPI1rlOGvZJEh71Ruz4SqljndUwOCoBUDRHRDU=@eagain.st/ fn must_namespace(caps: &client::Capabilities) -> bool { static MIN_GIT_VERSION_NAMESPACES: Lazy<Version> = - Lazy::new(|| Version::new("2.31.0").unwrap()); + Lazy::new(|| Version::new("2.25.0").unwrap()); remote_git_version(caps) .map(|version| version < *MIN_GIT_VERSION_NAMESPACES) -- 2.32.0 (Apple Git-132)
--- bins/Cargo.lock | 29 +++ bins/Cargo.toml | 1 + bins/tests/Cargo.toml | 13 + bins/tests/integration_test.rs | 449 +++++++++++++++++++++++++++++++++ link-git/src/protocol/fetch.rs | 5 +- link-git/src/protocol/ls.rs | 5 +- 6 files changed, 500 insertions(+), 2 deletions(-) create mode 100644 bins/tests/Cargo.toml create mode 100644 bins/tests/integration_test.rs diff --git a/bins/Cargo.lock b/bins/Cargo.lock index d5756144..4d0ea413 100644 --- a/bins/Cargo.lock @@ -838,6 +838,17 @@ dependencies = [ "termcolor", ] +[[package]] +name = "errno" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e2b2decb0484e15560df3210cf0d78654bb0864b2c138977c07e377a1bae0e2" +dependencies = [ + "kernel32-sys", + "libc", + "winapi 0.2.8", +] + [[package]] name = "event-listener" version = "2.5.2" @@ -2945,6 +2956,16 @@ dependencies = [ "human_format", ] +[[package]] +name = "pty" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f50f3d255966981eb4e4c5df3e983e6f7d163221f547406d83b6a460ff5c5ee8" +dependencies = [ + "errno", + "libc", +] + [[package]] name = "quanta" version = "0.4.1" @@ -3567,6 +3588,14 @@ dependencies = [ "winapi-util", ] +[[package]] +name = "tests" +version = "0.1.0" +dependencies = [ + "pty", + "serde_json", +] + [[package]] name = "textwrap" version = "0.15.0" diff --git a/bins/Cargo.toml b/bins/Cargo.toml index 3d1786f4..99b3c69f 100644 --- a/bins/Cargo.toml @@ -5,4 +5,5 @@ members = [ "lnk-gitd", "lnk-identities-dev", "lnk-profile-dev", + "tests", ] diff --git a/bins/tests/Cargo.toml b/bins/tests/Cargo.toml new file mode 100644 index 00000000..3ae82769 --- /dev/null @@ -0,0 +1,13 @@ +[package] +name = "tests" +version = "0.1.0" +authors = ["Han Xu <keepsimple@gmail.com>"] +edition = "2021" + +[dev-dependencies] +pty = "0.2" +serde_json = "1.0" + +[[test]] +name = "integration_test" +path = "integration_test.rs" diff --git a/bins/tests/integration_test.rs b/bins/tests/integration_test.rs new file mode 100644 index 00000000..770f1b5e --- /dev/null @@ -0,0 +1,449 @@ +// Copyright © 2022 The Radicle Link Contributors +// +// This file is part of radicle-link, distributed under the GPLv3 with Radicle +// Linking Exception. For full terms see the included LICENSE file. + +//! Integration test to exercise the programs in `bins`. + +use pty::fork::*; +use serde_json::{json, Value}; +use std::{ + env, + fs::File, + io::{BufRead, BufReader, Write}, + process::{Child, Command, Stdio}, + thread, + time::{Duration, SystemTime}, +}; + +/// This test is inspired by https://github.com/alexjg/linkd-playground +/// +/// Tests a typical scenario: there are two peer nodes and one seed node. +/// The main steps are: +/// - Setup a profile for each node in tmp home directories. +/// - Setup SSH keys for each profile. +/// - Create identities. +/// - Create a local repo for peer 1. +/// - Start `linkd` for the seed, and `lnk-gitd` for peer 1. +/// - Push peer 1 repo to its monorepo and to the seed. +/// - Clone the peer 1 repo to peer 2 via seed. +#[test] +fn two_peers_and_a_seed() { + let peer1_home = "/tmp/link-local-1"; + let peer2_home = "/tmp/link-local-2"; + let seed_home = "/tmp/seed-home"; + let passphrase = b"play\n"; + + println!("\n== create lnk homes for two peers and one seed ==\n"); + let (is_parent, _) = run_lnk(LnkCmd::ProfileCreate, peer1_home, passphrase); + if !is_parent { + return; + } + let (is_parent, _) = run_lnk(LnkCmd::ProfileCreate, peer2_home, passphrase); + if !is_parent { + return; + } + let (is_parent, _) = run_lnk(LnkCmd::ProfileCreate, seed_home, passphrase); + if !is_parent { + return; + } + + println!("\n== add ssh keys for each profile to the ssh-agent ==\n"); + let (is_parent, _) = run_lnk(LnkCmd::ProfileSshAdd, peer1_home, passphrase); + if !is_parent { + return; + } + let (is_parent, _) = run_lnk(LnkCmd::ProfileSshAdd, peer2_home, passphrase); + if !is_parent { + return; + } + let (is_parent, _) = run_lnk(LnkCmd::ProfileSshAdd, seed_home, passphrase); + if !is_parent { + return; + } + + println!("\n== Creating local link 1 identity ==\n"); + let peer1_name = "sockpuppet1".to_string(); + let (is_parent, output) = run_lnk(LnkCmd::IdPersonCreate(peer1_name), peer1_home, passphrase); + if !is_parent { + return; + } + let v: Value = serde_json::from_str(&output).unwrap(); + let urn1 = v["urn"].as_str().unwrap().to_string(); + let (is_parent, _) = run_lnk(LnkCmd::IdLocalSet(urn1), peer1_home, passphrase); + if !is_parent { + return; + } + + println!("\n== Creating local link 2 identity ==\n"); + let peer2_name = "sockpuppet2".to_string(); + let (is_parent, output) = run_lnk(LnkCmd::IdPersonCreate(peer2_name), peer2_home, passphrase); + if !is_parent { + return; + } + let v: Value = serde_json::from_str(&output).unwrap(); + let urn2 = v["urn"].as_str().unwrap().to_string(); + let (is_parent, _) = run_lnk(LnkCmd::IdLocalSet(urn2), peer2_home, passphrase); + if !is_parent { + return; + } + + println!("\n== Create a local repository ==\n"); + let peer1_proj = format!("peer1_proj_{}", timestamp()); + let (is_parent, output) = run_lnk( + LnkCmd::IdProjectCreate(peer1_proj.clone()), + peer1_home, + passphrase, + ); + if !is_parent { + return; + } + let v: Value = serde_json::from_str(&output).unwrap(); + let proj_urn = v["urn"].as_str().unwrap().to_string(); + println!("our project URN: {}", &proj_urn); + + println!("\n== Add the seed to the local peer seed configs ==\n"); + let (is_parent, seed_peer_id) = run_lnk(LnkCmd::ProfilePeer, seed_home, passphrase); + if !is_parent { + return; + } + let seed_endpoint = format!("{}@127.0.0.1:8799", &seed_peer_id); + + let (is_parent, peer1_profile) = run_lnk(LnkCmd::ProfileGet, peer1_home, passphrase); + if !is_parent { + return; + } + let peer1_seed = format!("{}/{}/seeds", peer1_home, peer1_profile); + let mut peer1_f = File::create(peer1_seed).unwrap(); + peer1_f.write_all(seed_endpoint.as_bytes()).unwrap(); + + let (is_parent, peer2_profile) = run_lnk(LnkCmd::ProfileGet, peer2_home, passphrase); + if !is_parent { + return; + } + let peer2_seed = format!("{}/{}/seeds", peer2_home, peer2_profile); + let mut peer2_f = File::create(peer2_seed).unwrap(); + peer2_f.write_all(seed_endpoint.as_bytes()).unwrap(); + + println!("\n== Start the seed linkd ==\n"); + let manifest_path = manifest_path(); + let mut linkd = spawn_linkd(seed_home, &manifest_path); + + println!("\n== Start the peer 1 gitd ==\n"); + let (is_parent, peer1_peer_id) = run_lnk(LnkCmd::ProfilePeer, peer1_home, passphrase); + if !is_parent { + return; + } + spawn_lnk_gitd(peer1_home, &manifest_path, &peer1_peer_id); + + println!("\n== Make some changes in the repo ==\n"); + env::set_current_dir(&peer1_proj).unwrap(); + let mut test_file = File::create("test").unwrap(); + test_file.write_all(b"test").unwrap(); + Command::new("git") + .arg("add") + .arg("test") + .output() + .expect("failed to do git add"); + let output = Command::new("git") + .arg("commit") + .arg("-m") + .arg("test commit") + .output() + .expect("failed to do git commit"); + println!("git-commit: {:?}", &output); + + println!("\n== Add the linkd remote to the repo ==\n"); + let remote_url = format!("ssh://rad@127.0.0.1:9987/{}.git", &proj_urn); + Command::new("git") + .arg("remote") + .arg("add") + .arg("linkd") + .arg(remote_url) + .output() + .expect("failed to do git remote add"); + + clean_up_known_hosts(); + + let is_parent = run_git_push(); + if !is_parent { + return; + } + + println!("\n== Clone to peer2 ==\n"); + + env::set_current_dir("..").unwrap(); // out of the peer1 proj directory. + let (is_parent, _) = run_lnk( + LnkCmd::Clone(proj_urn, peer1_peer_id), + peer2_home, + passphrase, + ); + if !is_parent { + return; + } + + println!("\n== Kill linkd (seed) ==\n"); + + linkd.kill().ok(); +} + +enum LnkCmd { + ProfileCreate, + ProfileGet, + ProfilePeer, + ProfileSshAdd, + IdPersonCreate(String), // the associated string is "the person's name". + IdLocalSet(String), // the associated string is "urn". + IdProjectCreate(String), // the associated string is "the project name". + Clone(String, String), // the associated string is "urn", "peer_id" +} + +/// Runs a `cmd` for `lnk_home`. Rebuilds `lnk` if necessary. +/// Return.0: true if this is the parent (i.e. test) process, +/// false if this is the child (i.e. lnk) process. +/// Return.1: an output that depends on the `cmd`. +fn run_lnk(cmd: LnkCmd, lnk_home: &str, passphrase: &[u8]) -> (bool, String) { + let fork = Fork::from_ptmx().unwrap(); + if let Some(mut parent) = fork.is_parent().ok() { + // Input the passphrase if necessary. + match cmd { + LnkCmd::ProfileCreate | LnkCmd::ProfileSshAdd => { + parent.write_all(passphrase).unwrap(); + println!("{}: wrote passphase", lnk_home); + }, + _ => {}, + } + + // Print the output and decode them if necessary. + let buf_reader = BufReader::new(parent); + let mut output = String::new(); + for line in buf_reader.lines() { + let line = line.unwrap(); + println!("{}: {}", lnk_home, line); + + match cmd { + LnkCmd::IdPersonCreate(ref _name) => { + if line.find("\"urn\":").is_some() { + output = line; // get the line with URN. + } + }, + LnkCmd::IdProjectCreate(ref _name) => { + if line.find("\"urn\":").is_some() { + output = line; // get the line with URN. + } + }, + LnkCmd::ProfileGet => { + output = line; // get the last line for profile id. + }, + LnkCmd::ProfilePeer => { + output = line; // get the last line for peer id. + }, + LnkCmd::Clone(ref _urn, ref _peer) => { + output = line; + }, + _ => {}, + } + } + + (true, output) + } else { + // Child process is to run `lnk`. + let manifest_path = manifest_path(); + + // cargo run \ + // --manifest-path $LINK_CHECKOUT/bins/Cargo.toml \ + // -p lnk -- "$@" + let mut lnk_cmd = Command::new("cargo"); + lnk_cmd + .env("LNK_HOME", lnk_home) + .arg("run") + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("lnk") + .arg("--"); + let full_cmd = match cmd { + LnkCmd::ProfileCreate => lnk_cmd.arg("profile").arg("create"), + LnkCmd::ProfileGet => lnk_cmd.arg("profile").arg("get"), + LnkCmd::ProfilePeer => lnk_cmd.arg("profile").arg("peer"), + LnkCmd::ProfileSshAdd => lnk_cmd.arg("profile").arg("ssh").arg("add"), + LnkCmd::IdPersonCreate(name) => { + let payload = json!({ "name": name }); + lnk_cmd + .arg("identities") + .arg("person") + .arg("create") + .arg("new") + .arg("--payload") + .arg(payload.to_string()) + }, + LnkCmd::IdLocalSet(urn) => lnk_cmd + .arg("identities") + .arg("local") + .arg("set") + .arg("--urn") + .arg(urn), + LnkCmd::IdProjectCreate(name) => { + let payload = json!({"name": name, "default_branch": "master"}); + let project_path = format!("./{}", name); + lnk_cmd + .arg("identities") + .arg("project") + .arg("create") + .arg("new") + .arg("--path") + .arg(project_path) + .arg("--payload") + .arg(payload.to_string()) + }, + LnkCmd::Clone(urn, peer_id) => { + let peer2_proj = format!("peer2_proj_{}", timestamp()); + lnk_cmd + .arg("clone") + .arg("--urn") + .arg(urn) + .arg("--path") + .arg(peer2_proj) + .arg("--peer") + .arg(peer_id) + }, + }; + full_cmd.status().expect("lnk cmd failed:"); + + (false, String::new()) + } +} + +fn spawn_linkd(lnk_home: &str, manifest_path: &str) -> Child { + let log_name = format!("linkd_{}.log", ×tamp()); + let log_file = File::create(&log_name).unwrap(); + let target_dir = bins_target_dir(); + let exec_path = format!("{}/debug/linkd", &target_dir); + + Command::new("cargo") + .arg("build") + .arg("--target-dir") + .arg(&target_dir) + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("linkd") + .output() + .expect("cargo build linkd failed"); + + let child = Command::new(&exec_path) + .env("RUST_BACKTRACE", "1") + .arg("--lnk-home") + .arg(lnk_home) + .arg("--track") + .arg("everything") + .arg("--protocol-listen") + .arg("127.0.0.1:8799") + .stdout(Stdio::from(log_file)) + .spawn() + .expect("linkd failed to start"); + println!("linkd stdout redirected to {}", &log_name); + thread::sleep(Duration::from_secs(1)); + child +} + +fn spawn_lnk_gitd(lnk_home: &str, manifest_path: &str, peer_id: &str) { + let port = "9987"; + let xdg_runtime_dir = env!("XDG_RUNTIME_DIR"); + let rpc_socket = format!("{}/link-peer-{}-rpc.socket", xdg_runtime_dir, peer_id); + let target_dir = bins_target_dir(); + let exec_path = format!("{}/debug/lnk-gitd", &target_dir); + + Command::new("cargo") + .arg("build") + .arg("--target-dir") + .arg(&target_dir) + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("lnk-gitd") + .stdout(Stdio::inherit()) + .output() + .expect("cargo build lnk-gitd failed"); + + Command::new("systemd-socket-activate") + .arg("-l") + .arg(port) + .arg("--fdname=ssh") + .arg("-E") + .arg("SSH_AUTH_SOCK") + .arg("-E") + .arg("RUST_BACKTRACE") + .arg(&exec_path) + .arg(lnk_home) + .arg("--linkd-rpc-socket") + .arg(rpc_socket) + .arg("--push-seeds") + .arg("--fetch-seeds") + .arg("--linger-timeout") + .arg("10000") + .spawn() + .expect("lnk-gitd failed to start"); + println!("started lnk-gitd"); + +} + +/// Returns true if this is the parent process, +/// returns false if this is the child process. +fn run_git_push() -> bool { + let fork = Fork::from_ptmx().unwrap(); + if let Some(mut parent) = fork.is_parent().ok() { + let yes = b"yes\n"; + let buf_reader = BufReader::new(parent); + for line in buf_reader.lines() { + let line = line.unwrap(); + println!("git-push: {}", line); + if line.find("key fingerprint").is_some() { + parent.write_all(yes).unwrap(); + } + } + + true + } else { + Command::new("git") + .arg("push") + .arg("linkd") + .status() + .expect("failed to do git push"); + false + } +} + +/// Returns UNIX_TIME in millis. +fn timestamp() -> u128 { + let now = SystemTime::now() + .duration_since(SystemTime::UNIX_EPOCH) + .unwrap(); + now.as_millis() +} + +/// Returns the full path of `bins` manifest file. +fn manifest_path() -> String { + let package_dir = env!("CARGO_MANIFEST_DIR"); + format!("{}/Cargo.toml", package_dir.strip_suffix("/tests").unwrap()) +} + +/// Returns the full path of `bins/target`. +fn bins_target_dir() -> String { + let package_dir = env!("CARGO_MANIFEST_DIR"); + format!("{}/target", package_dir.strip_suffix("/tests").unwrap()) +} + +fn clean_up_known_hosts() { + // ssh-keygen -f "/home/pi/.ssh/known_hosts" -R "[127.0.0.1]:9987" + let home_dir = env!("HOME"); + let known_hosts = format!("{}/.ssh/known_hosts", &home_dir); + let output = Command::new("ssh-keygen") + .arg("-f") + .arg(known_hosts) + .arg("-R") + .arg("[127.0.0.1]:9987") + .output() + .expect("failed to do ssh-keygen"); + println!("ssh-keygen: {:?}", &output); +} diff --git a/link-git/src/protocol/fetch.rs b/link-git/src/protocol/fetch.rs index a5a73065..98b92b51 100644 --- a/link-git/src/protocol/fetch.rs +++ b/link-git/src/protocol/fetch.rs @@ -38,8 +38,11 @@ use super::{packwriter::PackWriter, remote_git_version, transport}; // // cf. https://lore.kernel.org/git/CD2XNXHACAXS.13J6JTWZPO1JA@schmidt/ // Fixed in `git.git` 1ab13eb, which should land in 2.34 +// +// Based on testing with git 2.25.1 in Ubuntu 20.04, this workaround is +// not needed. Hence the checked version is lowered to 2.25.0. fn must_namespace_want_ref(caps: &client::Capabilities) -> bool { - static FIXED_AFTER: Lazy<Version> = Lazy::new(|| Version::new("2.33.0").unwrap()); + static FIXED_AFTER: Lazy<Version> = Lazy::new(|| Version::new("2.25.0").unwrap()); remote_git_version(caps) .map(|version| version <= *FIXED_AFTER) diff --git a/link-git/src/protocol/ls.rs b/link-git/src/protocol/ls.rs index b3516454..de95b4a3 100644 --- a/link-git/src/protocol/ls.rs +++ b/link-git/src/protocol/ls.rs @@ -22,9 +22,12 @@ use super::{remote_git_version, transport}; // Work around `git-upload-pack` not handling namespaces properly // // cf. https://lore.kernel.org/git/pMV5dJabxOBTD8kJBaPuWK0aS6OJhRQ7YFGwfhPCeSJEbPDrIFBza36nXBCgUCeUJWGmpjPI1rlOGvZJEh71Ruz4SqljndUwOCoBUDRHRDU=@eagain.st/ +// +// Based on testing with git 2.25.1 in Ubuntu 20.04, this workaround is +// not needed. Hence the checked version is lowered to 2.25.0. fn must_namespace(caps: &client::Capabilities) -> bool { static MIN_GIT_VERSION_NAMESPACES: Lazy<Version> = - Lazy::new(|| Version::new("2.31.0").unwrap()); + Lazy::new(|| Version::new("2.25.0").unwrap()); remote_git_version(caps) .map(|version| version < *MIN_GIT_VERSION_NAMESPACES) -- 2.32.0 (Apple Git-132)
--- bins/Cargo.lock | 29 +++ bins/Cargo.toml | 1 + bins/tests/Cargo.toml | 13 + bins/tests/integration_test.rs | 449 +++++++++++++++++++++++++++++++++ link-git/src/protocol/fetch.rs | 5 +- link-git/src/protocol/ls.rs | 5 +- 6 files changed, 500 insertions(+), 2 deletions(-) create mode 100644 bins/tests/Cargo.toml create mode 100644 bins/tests/integration_test.rs diff --git a/bins/Cargo.lock b/bins/Cargo.lock index d5756144..4d0ea413 100644 --- a/bins/Cargo.lock @@ -838,6 +838,17 @@ dependencies = [ "termcolor", ] +[[package]] +name = "errno" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e2b2decb0484e15560df3210cf0d78654bb0864b2c138977c07e377a1bae0e2" +dependencies = [ + "kernel32-sys", + "libc", + "winapi 0.2.8", +] + [[package]] name = "event-listener" version = "2.5.2" @@ -2945,6 +2956,16 @@ dependencies = [ "human_format", ] +[[package]] +name = "pty" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f50f3d255966981eb4e4c5df3e983e6f7d163221f547406d83b6a460ff5c5ee8" +dependencies = [ + "errno", + "libc", +] + [[package]] name = "quanta" version = "0.4.1" @@ -3567,6 +3588,14 @@ dependencies = [ "winapi-util", ] +[[package]] +name = "tests" +version = "0.1.0" +dependencies = [ + "pty", + "serde_json", +] + [[package]] name = "textwrap" version = "0.15.0" diff --git a/bins/Cargo.toml b/bins/Cargo.toml index 3d1786f4..99b3c69f 100644 --- a/bins/Cargo.toml @@ -5,4 +5,5 @@ members = [ "lnk-gitd", "lnk-identities-dev", "lnk-profile-dev", + "tests", ] diff --git a/bins/tests/Cargo.toml b/bins/tests/Cargo.toml new file mode 100644 index 00000000..3ae82769 --- /dev/null @@ -0,0 +1,13 @@ +[package] +name = "tests" +version = "0.1.0" +authors = ["Han Xu <keepsimple@gmail.com>"] +edition = "2021" + +[dev-dependencies] +pty = "0.2" +serde_json = "1.0" + +[[test]] +name = "integration_test" +path = "integration_test.rs" diff --git a/bins/tests/integration_test.rs b/bins/tests/integration_test.rs new file mode 100644 index 00000000..770f1b5e --- /dev/null @@ -0,0 +1,449 @@ +// Copyright © 2022 The Radicle Link Contributors +// +// This file is part of radicle-link, distributed under the GPLv3 with Radicle +// Linking Exception. For full terms see the included LICENSE file. + +//! Integration test to exercise the programs in `bins`. + +use pty::fork::*; +use serde_json::{json, Value}; +use std::{ + env, + fs::File, + io::{BufRead, BufReader, Write}, + process::{Child, Command, Stdio}, + thread, + time::{Duration, SystemTime}, +}; + +/// This test is inspired by https://github.com/alexjg/linkd-playground +/// +/// Tests a typical scenario: there are two peer nodes and one seed node. +/// The main steps are: +/// - Setup a profile for each node in tmp home directories. +/// - Setup SSH keys for each profile. +/// - Create identities. +/// - Create a local repo for peer 1. +/// - Start `linkd` for the seed, and `lnk-gitd` for peer 1. +/// - Push peer 1 repo to its monorepo and to the seed. +/// - Clone the peer 1 repo to peer 2 via seed. +#[test] +fn two_peers_and_a_seed() { + let peer1_home = "/tmp/link-local-1"; + let peer2_home = "/tmp/link-local-2"; + let seed_home = "/tmp/seed-home"; + let passphrase = b"play\n"; + + println!("\n== create lnk homes for two peers and one seed ==\n"); + let (is_parent, _) = run_lnk(LnkCmd::ProfileCreate, peer1_home, passphrase); + if !is_parent { + return; + } + let (is_parent, _) = run_lnk(LnkCmd::ProfileCreate, peer2_home, passphrase); + if !is_parent { + return; + } + let (is_parent, _) = run_lnk(LnkCmd::ProfileCreate, seed_home, passphrase); + if !is_parent { + return; + } + + println!("\n== add ssh keys for each profile to the ssh-agent ==\n"); + let (is_parent, _) = run_lnk(LnkCmd::ProfileSshAdd, peer1_home, passphrase); + if !is_parent { + return; + } + let (is_parent, _) = run_lnk(LnkCmd::ProfileSshAdd, peer2_home, passphrase); + if !is_parent { + return; + } + let (is_parent, _) = run_lnk(LnkCmd::ProfileSshAdd, seed_home, passphrase); + if !is_parent { + return; + } + + println!("\n== Creating local link 1 identity ==\n"); + let peer1_name = "sockpuppet1".to_string(); + let (is_parent, output) = run_lnk(LnkCmd::IdPersonCreate(peer1_name), peer1_home, passphrase); + if !is_parent { + return; + } + let v: Value = serde_json::from_str(&output).unwrap(); + let urn1 = v["urn"].as_str().unwrap().to_string(); + let (is_parent, _) = run_lnk(LnkCmd::IdLocalSet(urn1), peer1_home, passphrase); + if !is_parent { + return; + } + + println!("\n== Creating local link 2 identity ==\n"); + let peer2_name = "sockpuppet2".to_string(); + let (is_parent, output) = run_lnk(LnkCmd::IdPersonCreate(peer2_name), peer2_home, passphrase); + if !is_parent { + return; + } + let v: Value = serde_json::from_str(&output).unwrap(); + let urn2 = v["urn"].as_str().unwrap().to_string(); + let (is_parent, _) = run_lnk(LnkCmd::IdLocalSet(urn2), peer2_home, passphrase); + if !is_parent { + return; + } + + println!("\n== Create a local repository ==\n"); + let peer1_proj = format!("peer1_proj_{}", timestamp()); + let (is_parent, output) = run_lnk( + LnkCmd::IdProjectCreate(peer1_proj.clone()), + peer1_home, + passphrase, + ); + if !is_parent { + return; + } + let v: Value = serde_json::from_str(&output).unwrap(); + let proj_urn = v["urn"].as_str().unwrap().to_string(); + println!("our project URN: {}", &proj_urn); + + println!("\n== Add the seed to the local peer seed configs ==\n"); + let (is_parent, seed_peer_id) = run_lnk(LnkCmd::ProfilePeer, seed_home, passphrase); + if !is_parent { + return; + } + let seed_endpoint = format!("{}@127.0.0.1:8799", &seed_peer_id); + + let (is_parent, peer1_profile) = run_lnk(LnkCmd::ProfileGet, peer1_home, passphrase); + if !is_parent { + return; + } + let peer1_seed = format!("{}/{}/seeds", peer1_home, peer1_profile); + let mut peer1_f = File::create(peer1_seed).unwrap(); + peer1_f.write_all(seed_endpoint.as_bytes()).unwrap(); + + let (is_parent, peer2_profile) = run_lnk(LnkCmd::ProfileGet, peer2_home, passphrase); + if !is_parent { + return; + } + let peer2_seed = format!("{}/{}/seeds", peer2_home, peer2_profile); + let mut peer2_f = File::create(peer2_seed).unwrap(); + peer2_f.write_all(seed_endpoint.as_bytes()).unwrap(); + + println!("\n== Start the seed linkd ==\n"); + let manifest_path = manifest_path(); + let mut linkd = spawn_linkd(seed_home, &manifest_path); + + println!("\n== Start the peer 1 gitd ==\n"); + let (is_parent, peer1_peer_id) = run_lnk(LnkCmd::ProfilePeer, peer1_home, passphrase); + if !is_parent { + return; + } + spawn_lnk_gitd(peer1_home, &manifest_path, &peer1_peer_id); + + println!("\n== Make some changes in the repo ==\n"); + env::set_current_dir(&peer1_proj).unwrap(); + let mut test_file = File::create("test").unwrap(); + test_file.write_all(b"test").unwrap(); + Command::new("git") + .arg("add") + .arg("test") + .output() + .expect("failed to do git add"); + let output = Command::new("git") + .arg("commit") + .arg("-m") + .arg("test commit") + .output() + .expect("failed to do git commit"); + println!("git-commit: {:?}", &output); + + println!("\n== Add the linkd remote to the repo ==\n"); + let remote_url = format!("ssh://rad@127.0.0.1:9987/{}.git", &proj_urn); + Command::new("git") + .arg("remote") + .arg("add") + .arg("linkd") + .arg(remote_url) + .output() + .expect("failed to do git remote add"); + + clean_up_known_hosts(); + + let is_parent = run_git_push(); + if !is_parent { + return; + } + + println!("\n== Clone to peer2 ==\n"); + + env::set_current_dir("..").unwrap(); // out of the peer1 proj directory. + let (is_parent, _) = run_lnk( + LnkCmd::Clone(proj_urn, peer1_peer_id), + peer2_home, + passphrase, + ); + if !is_parent { + return; + } + + println!("\n== Kill linkd (seed) ==\n"); + + linkd.kill().ok(); +} + +enum LnkCmd { + ProfileCreate, + ProfileGet, + ProfilePeer, + ProfileSshAdd, + IdPersonCreate(String), // the associated string is "the person's name". + IdLocalSet(String), // the associated string is "urn". + IdProjectCreate(String), // the associated string is "the project name". + Clone(String, String), // the associated string is "urn", "peer_id" +} + +/// Runs a `cmd` for `lnk_home`. Rebuilds `lnk` if necessary. +/// Return.0: true if this is the parent (i.e. test) process, +/// false if this is the child (i.e. lnk) process. +/// Return.1: an output that depends on the `cmd`. +fn run_lnk(cmd: LnkCmd, lnk_home: &str, passphrase: &[u8]) -> (bool, String) { + let fork = Fork::from_ptmx().unwrap(); + if let Some(mut parent) = fork.is_parent().ok() { + // Input the passphrase if necessary. + match cmd { + LnkCmd::ProfileCreate | LnkCmd::ProfileSshAdd => { + parent.write_all(passphrase).unwrap(); + println!("{}: wrote passphase", lnk_home); + }, + _ => {}, + } + + // Print the output and decode them if necessary. + let buf_reader = BufReader::new(parent); + let mut output = String::new(); + for line in buf_reader.lines() { + let line = line.unwrap(); + println!("{}: {}", lnk_home, line); + + match cmd { + LnkCmd::IdPersonCreate(ref _name) => { + if line.find("\"urn\":").is_some() { + output = line; // get the line with URN. + } + }, + LnkCmd::IdProjectCreate(ref _name) => { + if line.find("\"urn\":").is_some() { + output = line; // get the line with URN. + } + }, + LnkCmd::ProfileGet => { + output = line; // get the last line for profile id. + }, + LnkCmd::ProfilePeer => { + output = line; // get the last line for peer id. + }, + LnkCmd::Clone(ref _urn, ref _peer) => { + output = line; + }, + _ => {}, + } + } + + (true, output) + } else { + // Child process is to run `lnk`. + let manifest_path = manifest_path(); + + // cargo run \ + // --manifest-path $LINK_CHECKOUT/bins/Cargo.toml \ + // -p lnk -- "$@" + let mut lnk_cmd = Command::new("cargo"); + lnk_cmd + .env("LNK_HOME", lnk_home) + .arg("run") + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("lnk") + .arg("--"); + let full_cmd = match cmd { + LnkCmd::ProfileCreate => lnk_cmd.arg("profile").arg("create"), + LnkCmd::ProfileGet => lnk_cmd.arg("profile").arg("get"), + LnkCmd::ProfilePeer => lnk_cmd.arg("profile").arg("peer"), + LnkCmd::ProfileSshAdd => lnk_cmd.arg("profile").arg("ssh").arg("add"), + LnkCmd::IdPersonCreate(name) => { + let payload = json!({ "name": name }); + lnk_cmd + .arg("identities") + .arg("person") + .arg("create") + .arg("new") + .arg("--payload") + .arg(payload.to_string()) + }, + LnkCmd::IdLocalSet(urn) => lnk_cmd + .arg("identities") + .arg("local") + .arg("set") + .arg("--urn") + .arg(urn), + LnkCmd::IdProjectCreate(name) => { + let payload = json!({"name": name, "default_branch": "master"}); + let project_path = format!("./{}", name); + lnk_cmd + .arg("identities") + .arg("project") + .arg("create") + .arg("new") + .arg("--path") + .arg(project_path) + .arg("--payload") + .arg(payload.to_string()) + }, + LnkCmd::Clone(urn, peer_id) => { + let peer2_proj = format!("peer2_proj_{}", timestamp()); + lnk_cmd + .arg("clone") + .arg("--urn") + .arg(urn) + .arg("--path") + .arg(peer2_proj) + .arg("--peer") + .arg(peer_id) + }, + }; + full_cmd.status().expect("lnk cmd failed:"); + + (false, String::new()) + } +} + +fn spawn_linkd(lnk_home: &str, manifest_path: &str) -> Child { + let log_name = format!("linkd_{}.log", ×tamp()); + let log_file = File::create(&log_name).unwrap(); + let target_dir = bins_target_dir(); + let exec_path = format!("{}/debug/linkd", &target_dir); + + Command::new("cargo") + .arg("build") + .arg("--target-dir") + .arg(&target_dir) + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("linkd") + .output() + .expect("cargo build linkd failed"); + + let child = Command::new(&exec_path) + .env("RUST_BACKTRACE", "1") + .arg("--lnk-home") + .arg(lnk_home) + .arg("--track") + .arg("everything") + .arg("--protocol-listen") + .arg("127.0.0.1:8799") + .stdout(Stdio::from(log_file)) + .spawn() + .expect("linkd failed to start"); + println!("linkd stdout redirected to {}", &log_name); + thread::sleep(Duration::from_secs(1)); + child +} + +fn spawn_lnk_gitd(lnk_home: &str, manifest_path: &str, peer_id: &str) { + let port = "9987"; + let xdg_runtime_dir = env!("XDG_RUNTIME_DIR"); + let rpc_socket = format!("{}/link-peer-{}-rpc.socket", xdg_runtime_dir, peer_id); + let target_dir = bins_target_dir(); + let exec_path = format!("{}/debug/lnk-gitd", &target_dir); + + Command::new("cargo") + .arg("build") + .arg("--target-dir") + .arg(&target_dir) + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("lnk-gitd") + .stdout(Stdio::inherit()) + .output() + .expect("cargo build lnk-gitd failed"); + + Command::new("systemd-socket-activate") + .arg("-l") + .arg(port) + .arg("--fdname=ssh") + .arg("-E") + .arg("SSH_AUTH_SOCK") + .arg("-E") + .arg("RUST_BACKTRACE") + .arg(&exec_path) + .arg(lnk_home) + .arg("--linkd-rpc-socket") + .arg(rpc_socket) + .arg("--push-seeds") + .arg("--fetch-seeds") + .arg("--linger-timeout") + .arg("10000") + .spawn() + .expect("lnk-gitd failed to start"); + println!("started lnk-gitd"); + +} +
Alex Good <alex@memoryandthought.me>This is platform dependent (i.e. OSX doesn't have `systemd-socket-activate`). We also don't really need to use the socket activated version of this, we could just run `lnk_home --push-seeds --fetch-seeds --addr 127.0.0.1:<some port>` (i.e. no `--linger-timeout`) which will immediately run `lnk-gitd` and keep it running until it's killed.
+/// Returns true if this is the parent process, +/// returns false if this is the child process. +fn run_git_push() -> bool { + let fork = Fork::from_ptmx().unwrap(); + if let Some(mut parent) = fork.is_parent().ok() { + let yes = b"yes\n"; + let buf_reader = BufReader::new(parent); + for line in buf_reader.lines() { + let line = line.unwrap(); + println!("git-push: {}", line); + if line.find("key fingerprint").is_some() { + parent.write_all(yes).unwrap(); + } + } + + true + } else { + Command::new("git") + .arg("push") + .arg("linkd") + .status() + .expect("failed to do git push"); + false + } +} + +/// Returns UNIX_TIME in millis. +fn timestamp() -> u128 { + let now = SystemTime::now() + .duration_since(SystemTime::UNIX_EPOCH) + .unwrap(); + now.as_millis() +} + +/// Returns the full path of `bins` manifest file. +fn manifest_path() -> String { + let package_dir = env!("CARGO_MANIFEST_DIR"); + format!("{}/Cargo.toml", package_dir.strip_suffix("/tests").unwrap()) +} + +/// Returns the full path of `bins/target`. +fn bins_target_dir() -> String { + let package_dir = env!("CARGO_MANIFEST_DIR"); + format!("{}/target", package_dir.strip_suffix("/tests").unwrap()) +} + +fn clean_up_known_hosts() { + // ssh-keygen -f "/home/pi/.ssh/known_hosts" -R "[127.0.0.1]:9987" + let home_dir = env!("HOME"); + let known_hosts = format!("{}/.ssh/known_hosts", &home_dir); + let output = Command::new("ssh-keygen") + .arg("-f") + .arg(known_hosts) + .arg("-R") + .arg("[127.0.0.1]:9987") + .output() + .expect("failed to do ssh-keygen"); + println!("ssh-keygen: {:?}", &output); +} diff --git a/link-git/src/protocol/fetch.rs b/link-git/src/protocol/fetch.rs index a5a73065..98b92b51 100644 --- a/link-git/src/protocol/fetch.rs +++ b/link-git/src/protocol/fetch.rs @@ -38,8 +38,11 @@ use super::{packwriter::PackWriter, remote_git_version, transport}; // // cf. https://lore.kernel.org/git/CD2XNXHACAXS.13J6JTWZPO1JA@schmidt/ // Fixed in `git.git` 1ab13eb, which should land in 2.34 +// +// Based on testing with git 2.25.1 in Ubuntu 20.04, this workaround is +// not needed. Hence the checked version is lowered to 2.25.0. fn must_namespace_want_ref(caps: &client::Capabilities) -> bool { - static FIXED_AFTER: Lazy<Version> = Lazy::new(|| Version::new("2.33.0").unwrap()); + static FIXED_AFTER: Lazy<Version> = Lazy::new(|| Version::new("2.25.0").unwrap()); remote_git_version(caps) .map(|version| version <= *FIXED_AFTER) diff --git a/link-git/src/protocol/ls.rs b/link-git/src/protocol/ls.rs index b3516454..de95b4a3 100644 --- a/link-git/src/protocol/ls.rs +++ b/link-git/src/protocol/ls.rs @@ -22,9 +22,12 @@ use super::{remote_git_version, transport}; // Work around `git-upload-pack` not handling namespaces properly // // cf. https://lore.kernel.org/git/pMV5dJabxOBTD8kJBaPuWK0aS6OJhRQ7YFGwfhPCeSJEbPDrIFBza36nXBCgUCeUJWGmpjPI1rlOGvZJEh71Ruz4SqljndUwOCoBUDRHRDU=@eagain.st/ +// +// Based on testing with git 2.25.1 in Ubuntu 20.04, this workaround is +// not needed. Hence the checked version is lowered to 2.25.0. fn must_namespace(caps: &client::Capabilities) -> bool { static MIN_GIT_VERSION_NAMESPACES: Lazy<Version> = - Lazy::new(|| Version::new("2.31.0").unwrap()); + Lazy::new(|| Version::new("2.25.0").unwrap()); remote_git_version(caps) .map(|version| version < *MIN_GIT_VERSION_NAMESPACES) -- 2.32.0 (Apple Git-132)
Alex Good <alex@memoryandthought.me>I still find this return and switch on the returned `bool` a bit cumbersome. Is it worth using something like https://docs.rs/pty-process/latest/pty_process/struct.Child.html which would give us basically the same API as `std::process::Child`? Failing that we could write a little macro to remove all the `is_parent` checks in the calling code.
--- bins/Cargo.lock | 29 +++ bins/Cargo.toml | 1 + bins/tests/Cargo.toml | 13 + bins/tests/integration_test.rs | 449 +++++++++++++++++++++++++++++++++ link-git/src/protocol/fetch.rs | 5 +- link-git/src/protocol/ls.rs | 5 +- 6 files changed, 500 insertions(+), 2 deletions(-) create mode 100644 bins/tests/Cargo.toml create mode 100644 bins/tests/integration_test.rs diff --git a/bins/Cargo.lock b/bins/Cargo.lock index d5756144..4d0ea413 100644 --- a/bins/Cargo.lock @@ -838,6 +838,17 @@ dependencies = [ "termcolor", ] +[[package]] +name = "errno" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e2b2decb0484e15560df3210cf0d78654bb0864b2c138977c07e377a1bae0e2" +dependencies = [ + "kernel32-sys", + "libc", + "winapi 0.2.8", +] + [[package]] name = "event-listener" version = "2.5.2" @@ -2945,6 +2956,16 @@ dependencies = [ "human_format", ] +[[package]] +name = "pty" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f50f3d255966981eb4e4c5df3e983e6f7d163221f547406d83b6a460ff5c5ee8" +dependencies = [ + "errno", + "libc", +] + [[package]] name = "quanta" version = "0.4.1" @@ -3567,6 +3588,14 @@ dependencies = [ "winapi-util", ] +[[package]] +name = "tests" +version = "0.1.0" +dependencies = [ + "pty", + "serde_json", +] + [[package]] name = "textwrap" version = "0.15.0" diff --git a/bins/Cargo.toml b/bins/Cargo.toml index 3d1786f4..99b3c69f 100644 --- a/bins/Cargo.toml @@ -5,4 +5,5 @@ members = [ "lnk-gitd", "lnk-identities-dev", "lnk-profile-dev", + "tests", ] diff --git a/bins/tests/Cargo.toml b/bins/tests/Cargo.toml new file mode 100644 index 00000000..3ae82769 --- /dev/null @@ -0,0 +1,13 @@ +[package] +name = "tests" +version = "0.1.0" +authors = ["Han Xu <keepsimple@gmail.com>"] +edition = "2021" + +[dev-dependencies] +pty = "0.2" +serde_json = "1.0" + +[[test]] +name = "integration_test" +path = "integration_test.rs" diff --git a/bins/tests/integration_test.rs b/bins/tests/integration_test.rs new file mode 100644 index 00000000..770f1b5e --- /dev/null @@ -0,0 +1,449 @@ +// Copyright © 2022 The Radicle Link Contributors +// +// This file is part of radicle-link, distributed under the GPLv3 with Radicle +// Linking Exception. For full terms see the included LICENSE file. + +//! Integration test to exercise the programs in `bins`. + +use pty::fork::*; +use serde_json::{json, Value}; +use std::{ + env, + fs::File, + io::{BufRead, BufReader, Write}, + process::{Child, Command, Stdio}, + thread, + time::{Duration, SystemTime}, +}; + +/// This test is inspired by https://github.com/alexjg/linkd-playground +/// +/// Tests a typical scenario: there are two peer nodes and one seed node. +/// The main steps are: +/// - Setup a profile for each node in tmp home directories. +/// - Setup SSH keys for each profile. +/// - Create identities. +/// - Create a local repo for peer 1. +/// - Start `linkd` for the seed, and `lnk-gitd` for peer 1. +/// - Push peer 1 repo to its monorepo and to the seed. +/// - Clone the peer 1 repo to peer 2 via seed. +#[test] +fn two_peers_and_a_seed() { + let peer1_home = "/tmp/link-local-1"; + let peer2_home = "/tmp/link-local-2"; + let seed_home = "/tmp/seed-home"; + let passphrase = b"play\n"; + + println!("\n== create lnk homes for two peers and one seed ==\n"); + let (is_parent, _) = run_lnk(LnkCmd::ProfileCreate, peer1_home, passphrase); + if !is_parent { + return; + } + let (is_parent, _) = run_lnk(LnkCmd::ProfileCreate, peer2_home, passphrase); + if !is_parent { + return; + } + let (is_parent, _) = run_lnk(LnkCmd::ProfileCreate, seed_home, passphrase); + if !is_parent { + return; + } + + println!("\n== add ssh keys for each profile to the ssh-agent ==\n"); + let (is_parent, _) = run_lnk(LnkCmd::ProfileSshAdd, peer1_home, passphrase); + if !is_parent { + return; + } + let (is_parent, _) = run_lnk(LnkCmd::ProfileSshAdd, peer2_home, passphrase); + if !is_parent { + return; + } + let (is_parent, _) = run_lnk(LnkCmd::ProfileSshAdd, seed_home, passphrase); + if !is_parent { + return; + } + + println!("\n== Creating local link 1 identity ==\n"); + let peer1_name = "sockpuppet1".to_string(); + let (is_parent, output) = run_lnk(LnkCmd::IdPersonCreate(peer1_name), peer1_home, passphrase); + if !is_parent { + return; + } + let v: Value = serde_json::from_str(&output).unwrap(); + let urn1 = v["urn"].as_str().unwrap().to_string(); + let (is_parent, _) = run_lnk(LnkCmd::IdLocalSet(urn1), peer1_home, passphrase); + if !is_parent { + return; + } + + println!("\n== Creating local link 2 identity ==\n"); + let peer2_name = "sockpuppet2".to_string(); + let (is_parent, output) = run_lnk(LnkCmd::IdPersonCreate(peer2_name), peer2_home, passphrase); + if !is_parent { + return; + } + let v: Value = serde_json::from_str(&output).unwrap(); + let urn2 = v["urn"].as_str().unwrap().to_string(); + let (is_parent, _) = run_lnk(LnkCmd::IdLocalSet(urn2), peer2_home, passphrase); + if !is_parent { + return; + } + + println!("\n== Create a local repository ==\n"); + let peer1_proj = format!("peer1_proj_{}", timestamp()); + let (is_parent, output) = run_lnk( + LnkCmd::IdProjectCreate(peer1_proj.clone()), + peer1_home, + passphrase, + ); + if !is_parent { + return; + } + let v: Value = serde_json::from_str(&output).unwrap(); + let proj_urn = v["urn"].as_str().unwrap().to_string(); + println!("our project URN: {}", &proj_urn); + + println!("\n== Add the seed to the local peer seed configs ==\n"); + let (is_parent, seed_peer_id) = run_lnk(LnkCmd::ProfilePeer, seed_home, passphrase); + if !is_parent { + return; + } + let seed_endpoint = format!("{}@127.0.0.1:8799", &seed_peer_id); + + let (is_parent, peer1_profile) = run_lnk(LnkCmd::ProfileGet, peer1_home, passphrase); + if !is_parent { + return; + } + let peer1_seed = format!("{}/{}/seeds", peer1_home, peer1_profile); + let mut peer1_f = File::create(peer1_seed).unwrap(); + peer1_f.write_all(seed_endpoint.as_bytes()).unwrap(); + + let (is_parent, peer2_profile) = run_lnk(LnkCmd::ProfileGet, peer2_home, passphrase); + if !is_parent { + return; + } + let peer2_seed = format!("{}/{}/seeds", peer2_home, peer2_profile); + let mut peer2_f = File::create(peer2_seed).unwrap(); + peer2_f.write_all(seed_endpoint.as_bytes()).unwrap(); + + println!("\n== Start the seed linkd ==\n"); + let manifest_path = manifest_path(); + let mut linkd = spawn_linkd(seed_home, &manifest_path); + + println!("\n== Start the peer 1 gitd ==\n"); + let (is_parent, peer1_peer_id) = run_lnk(LnkCmd::ProfilePeer, peer1_home, passphrase); + if !is_parent { + return; + } + spawn_lnk_gitd(peer1_home, &manifest_path, &peer1_peer_id); + + println!("\n== Make some changes in the repo ==\n"); + env::set_current_dir(&peer1_proj).unwrap(); + let mut test_file = File::create("test").unwrap(); + test_file.write_all(b"test").unwrap(); + Command::new("git") + .arg("add") + .arg("test") + .output() + .expect("failed to do git add"); + let output = Command::new("git") + .arg("commit") + .arg("-m") + .arg("test commit") + .output() + .expect("failed to do git commit"); + println!("git-commit: {:?}", &output); + + println!("\n== Add the linkd remote to the repo ==\n"); + let remote_url = format!("ssh://rad@127.0.0.1:9987/{}.git", &proj_urn); + Command::new("git") + .arg("remote") + .arg("add") + .arg("linkd") + .arg(remote_url) + .output() + .expect("failed to do git remote add"); + + clean_up_known_hosts(); + + let is_parent = run_git_push(); + if !is_parent { + return; + } + + println!("\n== Clone to peer2 ==\n"); + + env::set_current_dir("..").unwrap(); // out of the peer1 proj directory. + let (is_parent, _) = run_lnk( + LnkCmd::Clone(proj_urn, peer1_peer_id), + peer2_home, + passphrase, + ); + if !is_parent { + return; + } + + println!("\n== Kill linkd (seed) ==\n"); + + linkd.kill().ok(); +} + +enum LnkCmd { + ProfileCreate, + ProfileGet, + ProfilePeer, + ProfileSshAdd, + IdPersonCreate(String), // the associated string is "the person's name". + IdLocalSet(String), // the associated string is "urn". + IdProjectCreate(String), // the associated string is "the project name". + Clone(String, String), // the associated string is "urn", "peer_id" +} + +/// Runs a `cmd` for `lnk_home`. Rebuilds `lnk` if necessary. +/// Return.0: true if this is the parent (i.e. test) process, +/// false if this is the child (i.e. lnk) process. +/// Return.1: an output that depends on the `cmd`. +fn run_lnk(cmd: LnkCmd, lnk_home: &str, passphrase: &[u8]) -> (bool, String) { + let fork = Fork::from_ptmx().unwrap(); + if let Some(mut parent) = fork.is_parent().ok() { + // Input the passphrase if necessary. + match cmd { + LnkCmd::ProfileCreate | LnkCmd::ProfileSshAdd => { + parent.write_all(passphrase).unwrap(); + println!("{}: wrote passphase", lnk_home); + }, + _ => {}, + } + + // Print the output and decode them if necessary. + let buf_reader = BufReader::new(parent); + let mut output = String::new(); + for line in buf_reader.lines() { + let line = line.unwrap(); + println!("{}: {}", lnk_home, line); + + match cmd { + LnkCmd::IdPersonCreate(ref _name) => { + if line.find("\"urn\":").is_some() { + output = line; // get the line with URN. + } + }, + LnkCmd::IdProjectCreate(ref _name) => { + if line.find("\"urn\":").is_some() { + output = line; // get the line with URN. + } + }, + LnkCmd::ProfileGet => { + output = line; // get the last line for profile id. + }, + LnkCmd::ProfilePeer => { + output = line; // get the last line for peer id. + }, + LnkCmd::Clone(ref _urn, ref _peer) => { + output = line; + }, + _ => {}, + } + } + + (true, output) + } else { + // Child process is to run `lnk`. + let manifest_path = manifest_path(); + + // cargo run \ + // --manifest-path $LINK_CHECKOUT/bins/Cargo.toml \ + // -p lnk -- "$@" + let mut lnk_cmd = Command::new("cargo"); + lnk_cmd + .env("LNK_HOME", lnk_home) + .arg("run") + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("lnk") + .arg("--"); + let full_cmd = match cmd { + LnkCmd::ProfileCreate => lnk_cmd.arg("profile").arg("create"), + LnkCmd::ProfileGet => lnk_cmd.arg("profile").arg("get"), + LnkCmd::ProfilePeer => lnk_cmd.arg("profile").arg("peer"), + LnkCmd::ProfileSshAdd => lnk_cmd.arg("profile").arg("ssh").arg("add"), + LnkCmd::IdPersonCreate(name) => { + let payload = json!({ "name": name }); + lnk_cmd + .arg("identities") + .arg("person") + .arg("create") + .arg("new") + .arg("--payload") + .arg(payload.to_string()) + }, + LnkCmd::IdLocalSet(urn) => lnk_cmd + .arg("identities") + .arg("local") + .arg("set") + .arg("--urn") + .arg(urn), + LnkCmd::IdProjectCreate(name) => { + let payload = json!({"name": name, "default_branch": "master"}); + let project_path = format!("./{}", name); + lnk_cmd + .arg("identities") + .arg("project") + .arg("create") + .arg("new") + .arg("--path") + .arg(project_path) + .arg("--payload") + .arg(payload.to_string()) + }, + LnkCmd::Clone(urn, peer_id) => { + let peer2_proj = format!("peer2_proj_{}", timestamp()); + lnk_cmd + .arg("clone") + .arg("--urn") + .arg(urn) + .arg("--path") + .arg(peer2_proj) + .arg("--peer") + .arg(peer_id) + }, + }; + full_cmd.status().expect("lnk cmd failed:"); + + (false, String::new()) + } +} + +fn spawn_linkd(lnk_home: &str, manifest_path: &str) -> Child { + let log_name = format!("linkd_{}.log", ×tamp()); + let log_file = File::create(&log_name).unwrap(); + let target_dir = bins_target_dir(); + let exec_path = format!("{}/debug/linkd", &target_dir); + + Command::new("cargo") + .arg("build") + .arg("--target-dir") + .arg(&target_dir) + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("linkd") + .output() + .expect("cargo build linkd failed"); + + let child = Command::new(&exec_path) + .env("RUST_BACKTRACE", "1") + .arg("--lnk-home") + .arg(lnk_home) + .arg("--track") + .arg("everything") + .arg("--protocol-listen") + .arg("127.0.0.1:8799") + .stdout(Stdio::from(log_file)) + .spawn() + .expect("linkd failed to start"); + println!("linkd stdout redirected to {}", &log_name); + thread::sleep(Duration::from_secs(1)); + child +} + +fn spawn_lnk_gitd(lnk_home: &str, manifest_path: &str, peer_id: &str) { + let port = "9987"; + let xdg_runtime_dir = env!("XDG_RUNTIME_DIR"); + let rpc_socket = format!("{}/link-peer-{}-rpc.socket", xdg_runtime_dir, peer_id); + let target_dir = bins_target_dir(); + let exec_path = format!("{}/debug/lnk-gitd", &target_dir); + + Command::new("cargo") + .arg("build") + .arg("--target-dir") + .arg(&target_dir) + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("lnk-gitd") + .stdout(Stdio::inherit()) + .output() + .expect("cargo build lnk-gitd failed"); + + Command::new("systemd-socket-activate") + .arg("-l") + .arg(port) + .arg("--fdname=ssh") + .arg("-E") + .arg("SSH_AUTH_SOCK") + .arg("-E") + .arg("RUST_BACKTRACE") + .arg(&exec_path) + .arg(lnk_home) + .arg("--linkd-rpc-socket") + .arg(rpc_socket) + .arg("--push-seeds") + .arg("--fetch-seeds") + .arg("--linger-timeout") + .arg("10000") + .spawn() + .expect("lnk-gitd failed to start"); + println!("started lnk-gitd"); + +} + +/// Returns true if this is the parent process, +/// returns false if this is the child process. +fn run_git_push() -> bool { + let fork = Fork::from_ptmx().unwrap(); + if let Some(mut parent) = fork.is_parent().ok() { + let yes = b"yes\n"; + let buf_reader = BufReader::new(parent); + for line in buf_reader.lines() { + let line = line.unwrap(); + println!("git-push: {}", line); + if line.find("key fingerprint").is_some() { + parent.write_all(yes).unwrap(); + } + } + + true + } else { + Command::new("git") + .arg("push") + .arg("linkd") + .status() + .expect("failed to do git push"); + false + } +} + +/// Returns UNIX_TIME in millis. +fn timestamp() -> u128 { + let now = SystemTime::now() + .duration_since(SystemTime::UNIX_EPOCH) + .unwrap(); + now.as_millis() +} + +/// Returns the full path of `bins` manifest file. +fn manifest_path() -> String { + let package_dir = env!("CARGO_MANIFEST_DIR"); + format!("{}/Cargo.toml", package_dir.strip_suffix("/tests").unwrap()) +} + +/// Returns the full path of `bins/target`. +fn bins_target_dir() -> String { + let package_dir = env!("CARGO_MANIFEST_DIR"); + format!("{}/target", package_dir.strip_suffix("/tests").unwrap()) +} + +fn clean_up_known_hosts() { + // ssh-keygen -f "/home/pi/.ssh/known_hosts" -R "[127.0.0.1]:9987" + let home_dir = env!("HOME"); + let known_hosts = format!("{}/.ssh/known_hosts", &home_dir); + let output = Command::new("ssh-keygen") + .arg("-f") + .arg(known_hosts) + .arg("-R") + .arg("[127.0.0.1]:9987") + .output() + .expect("failed to do ssh-keygen"); + println!("ssh-keygen: {:?}", &output); +} diff --git a/link-git/src/protocol/fetch.rs b/link-git/src/protocol/fetch.rs index a5a73065..98b92b51 100644 --- a/link-git/src/protocol/fetch.rs +++ b/link-git/src/protocol/fetch.rs @@ -38,8 +38,11 @@ use super::{packwriter::PackWriter, remote_git_version, transport}; // // cf. https://lore.kernel.org/git/CD2XNXHACAXS.13J6JTWZPO1JA@schmidt/ // Fixed in `git.git` 1ab13eb, which should land in 2.34 +// +// Based on testing with git 2.25.1 in Ubuntu 20.04, this workaround is +// not needed. Hence the checked version is lowered to 2.25.0. fn must_namespace_want_ref(caps: &client::Capabilities) -> bool { - static FIXED_AFTER: Lazy<Version> = Lazy::new(|| Version::new("2.33.0").unwrap()); + static FIXED_AFTER: Lazy<Version> = Lazy::new(|| Version::new("2.25.0").unwrap()); remote_git_version(caps) .map(|version| version <= *FIXED_AFTER) diff --git a/link-git/src/protocol/ls.rs b/link-git/src/protocol/ls.rs index b3516454..de95b4a3 100644 --- a/link-git/src/protocol/ls.rs +++ b/link-git/src/protocol/ls.rs @@ -22,9 +22,12 @@ use super::{remote_git_version, transport}; // Work around `git-upload-pack` not handling namespaces properly // // cf. https://lore.kernel.org/git/pMV5dJabxOBTD8kJBaPuWK0aS6OJhRQ7YFGwfhPCeSJEbPDrIFBza36nXBCgUCeUJWGmpjPI1rlOGvZJEh71Ruz4SqljndUwOCoBUDRHRDU=@eagain.st/ +// +// Based on testing with git 2.25.1 in Ubuntu 20.04, this workaround is +// not needed. Hence the checked version is lowered to 2.25.0. fn must_namespace(caps: &client::Capabilities) -> bool { static MIN_GIT_VERSION_NAMESPACES: Lazy<Version> = - Lazy::new(|| Version::new("2.31.0").unwrap()); + Lazy::new(|| Version::new("2.25.0").unwrap()); remote_git_version(caps) .map(|version| version < *MIN_GIT_VERSION_NAMESPACES) -- 2.32.0 (Apple Git-132)
--- bins/Cargo.lock | 29 +++ bins/Cargo.toml | 1 + bins/tests/Cargo.toml | 13 + bins/tests/integration_test.rs | 456 +++++++++++++++++++++++++++++++++ link-git/src/protocol/fetch.rs | 5 +- link-git/src/protocol/ls.rs | 5 +- 6 files changed, 507 insertions(+), 2 deletions(-) create mode 100644 bins/tests/Cargo.toml create mode 100644 bins/tests/integration_test.rs diff --git a/bins/Cargo.lock b/bins/Cargo.lock index d5756144..4d0ea413 100644 --- a/bins/Cargo.lock @@ -838,6 +838,17 @@ dependencies = [ "termcolor", ] +[[package]] +name = "errno" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e2b2decb0484e15560df3210cf0d78654bb0864b2c138977c07e377a1bae0e2" +dependencies = [ + "kernel32-sys", + "libc", + "winapi 0.2.8", +] + [[package]] name = "event-listener" version = "2.5.2" @@ -2945,6 +2956,16 @@ dependencies = [ "human_format", ] +[[package]] +name = "pty" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f50f3d255966981eb4e4c5df3e983e6f7d163221f547406d83b6a460ff5c5ee8" +dependencies = [ + "errno", + "libc", +] + [[package]] name = "quanta" version = "0.4.1" @@ -3567,6 +3588,14 @@ dependencies = [ "winapi-util", ] +[[package]] +name = "tests" +version = "0.1.0" +dependencies = [ + "pty", + "serde_json", +] + [[package]] name = "textwrap" version = "0.15.0" diff --git a/bins/Cargo.toml b/bins/Cargo.toml index 3d1786f4..99b3c69f 100644 --- a/bins/Cargo.toml @@ -5,4 +5,5 @@ members = [ "lnk-gitd", "lnk-identities-dev", "lnk-profile-dev", + "tests", ] diff --git a/bins/tests/Cargo.toml b/bins/tests/Cargo.toml new file mode 100644 index 00000000..3ae82769 --- /dev/null @@ -0,0 +1,13 @@ +[package] +name = "tests" +version = "0.1.0" +authors = ["Han Xu <keepsimple@gmail.com>"] +edition = "2021" + +[dev-dependencies] +pty = "0.2" +serde_json = "1.0" + +[[test]] +name = "integration_test" +path = "integration_test.rs" diff --git a/bins/tests/integration_test.rs b/bins/tests/integration_test.rs new file mode 100644 index 00000000..2d2a4a92 --- /dev/null @@ -0,0 +1,456 @@ +// Copyright © 2022 The Radicle Link Contributors +// +// This file is part of radicle-link, distributed under the GPLv3 with Radicle +// Linking Exception. For full terms see the included LICENSE file. + +//! Integration test to exercise the programs in `bins`. + +use pty::fork::*; +use serde_json::{json, Value}; +use std::{ + env, + fs::File, + io::{BufRead, BufReader, Write}, + process::{Child, Command, Stdio}, + thread, + time::{Duration, SystemTime}, +}; + +/// This test is inspired by https://github.com/alexjg/linkd-playground +/// +/// Tests a typical scenario: there are two peer nodes and one seed node. +/// The main steps are: +/// - Setup a profile for each node in tmp home directories. +/// - Setup SSH keys for each profile. +/// - Create identities. +/// - Create a local repo for peer 1. +/// - Start `linkd` for the seed, and `lnk-gitd` for peer 1. +/// - Push peer 1 repo to its monorepo and to the seed. +/// - Clone the peer 1 repo to peer 2 via seed. +/// +/// This test only works in Linux at this moment. +#[test] +fn two_peers_and_a_seed() { + let peer1_home = "/tmp/link-local-1"; + let peer2_home = "/tmp/link-local-2"; + let seed_home = "/tmp/seed-home"; + let passphrase = b"play\n"; + + { + println!("\n== create lnk homes for two peers and one seed ==\n"); + let cmd = LnkCmd::ProfileCreate; + run_lnk!(&cmd, peer1_home, passphrase); + run_lnk!(&cmd, peer2_home, passphrase); + run_lnk!(&cmd, seed_home, passphrase); + } + + { + println!("\n== add ssh keys for each profile to the ssh-agent ==\n"); + let cmd = LnkCmd::ProfileSshAdd; + run_lnk!(&cmd, peer1_home, passphrase); + run_lnk!(&cmd, peer2_home, passphrase); + run_lnk!(&cmd, seed_home, passphrase); + } + + { + println!("\n== Creating local link 1 identity ==\n"); + let peer1_name = "sockpuppet1".to_string(); + let cmd = LnkCmd::IdPersonCreate { name: peer1_name }; + let output = run_lnk!(&cmd, peer1_home, passphrase); + let v: Value = serde_json::from_str(&output).unwrap(); + let urn = v["urn"].as_str().unwrap().to_string(); + let cmd = LnkCmd::IdLocalSet { urn }; + run_lnk!(&cmd, peer1_home, passphrase); + } + + { + println!("\n== Creating local link 2 identity ==\n"); + let peer2_name = "sockpuppet2".to_string(); + let cmd = LnkCmd::IdPersonCreate { name: peer2_name }; + let output = run_lnk!(&cmd, peer2_home, passphrase); + let v: Value = serde_json::from_str(&output).unwrap(); + let urn = v["urn"].as_str().unwrap().to_string(); + let cmd = LnkCmd::IdLocalSet { urn }; + run_lnk!(&cmd, peer2_home, passphrase); + } + + let peer1_proj_dir = format!("peer1_proj_{}", timestamp()); + let peer1_proj_urn = { + println!("\n== Create a local repository for peer1 ==\n"); + let cmd = LnkCmd::IdProjectCreate { + name: peer1_proj_dir.clone(), + }; + let output = run_lnk!(&cmd, peer1_home, passphrase); + let v: Value = serde_json::from_str(&output).unwrap(); + let proj_urn = v["urn"].as_str().unwrap().to_string(); + println!("our project URN: {}", &proj_urn); + proj_urn + }; + + println!("\n== Add the seed to the local peer seed configs ==\n"); + let seed_endpoint = { + let cmd = LnkCmd::ProfilePeer; + let seed_peer_id = run_lnk!(&cmd, seed_home, passphrase); + format!("{}@127.0.0.1:8799", &seed_peer_id) + }; + + { + // Create seed file for peer1 + let cmd = LnkCmd::ProfileGet; + let peer1_profile = run_lnk!(&cmd, peer1_home, passphrase); + let peer1_seed = format!("{}/{}/seeds", peer1_home, peer1_profile); + let mut peer1_seed_f = File::create(peer1_seed).unwrap(); + peer1_seed_f.write_all(seed_endpoint.as_bytes()).unwrap(); + } + + { + // Create seed file for peer2 + let cmd = LnkCmd::ProfileGet; + let peer2_profile = run_lnk!(&cmd, peer2_home, passphrase); + let peer2_seed = format!("{}/{}/seeds", peer2_home, peer2_profile); + let mut peer2_seed_f = File::create(peer2_seed).unwrap(); + peer2_seed_f.write_all(seed_endpoint.as_bytes()).unwrap(); + } + + println!("\n== Start the seed linkd ==\n"); + let manifest_path = manifest_path(); + let mut linkd = spawn_linkd(seed_home, &manifest_path); + + println!("\n== Start the peer 1 gitd ==\n"); + let cmd = LnkCmd::ProfilePeer; + let gitd_addr = "127.0.0.1:9987"; + let peer1_peer_id = run_lnk!(&cmd, peer1_home, passphrase); + let mut gitd = spawn_lnk_gitd(peer1_home, &manifest_path, gitd_addr); + + println!("\n== Make some changes in the repo: add and commit a test file ==\n"); + env::set_current_dir(&peer1_proj_dir).unwrap(); + let mut test_file = File::create("test").unwrap(); + test_file.write_all(b"test").unwrap(); + Command::new("git") + .arg("add") + .arg("test") + .output() + .expect("failed to do git add"); + let output = Command::new("git") + .arg("commit") + .arg("-m") + .arg("test commit") + .output() + .expect("failed to do git commit"); + println!("git-commit: {:?}", &output); + + println!("\n== Add the linkd remote to the repo ==\n"); + let remote_url = format!("ssh://rad@{}/{}.git", gitd_addr, &peer1_proj_urn); + Command::new("git") + .arg("remote") + .arg("add") + .arg("linkd") + .arg(remote_url) + .output() + .expect("failed to do git remote add"); + + clean_up_known_hosts(); + + if run_git_push_in_child_process() { + // The child process is done with git push. + return; + } + + let peer1_last_commit = git_last_commit(); + + println!("\n== Clone to peer2 ==\n"); + + env::set_current_dir("..").unwrap(); // out of the peer1 proj directory. + let peer2_proj = format!("peer2_proj_{}", timestamp()); + let cmd = LnkCmd::Clone { + urn: peer1_proj_urn, + peer_id: peer1_peer_id, + path: peer2_proj.clone(), + }; + run_lnk!(&cmd, peer2_home, passphrase); + + env::set_current_dir(peer2_proj).unwrap(); + let peer2_last_commit = git_last_commit(); + println!("\n== peer1 proj last commit: {}", &peer1_last_commit); + println!("\n== peer2 proj last commit: {}", &peer2_last_commit); + + println!("\n== Cleanup: kill linkd (seed) and gitd (peer1) ==\n"); + + linkd.kill().ok(); + gitd.kill().ok(); + + assert_eq!(peer1_last_commit, peer2_last_commit); +} + +enum LnkCmd { + ProfileCreate, + ProfileGet, + ProfilePeer, + ProfileSshAdd, + IdPersonCreate { + name: String, + }, + IdLocalSet { + urn: String, + }, + IdProjectCreate { + name: String, + }, + Clone { + urn: String, + peer_id: String, + path: String, + }, +} + +/// Runs a `lnk` command of `$cmd` using `$lnk_home` as the node home. +/// Also support the passphrase input for commands that need it. +#[macro_export] +macro_rules! run_lnk { + ( $cmd:expr, $lnk_home:ident, $passphrase:ident ) => {{ + let fork = Fork::from_ptmx().unwrap(); + if let Some(mut parent) = fork.is_parent().ok() { + match $cmd { + LnkCmd::ProfileCreate | LnkCmd::ProfileSshAdd => { + // Input the passphrase if necessary. + parent.write_all($passphrase).unwrap(); + }, + _ => {}, + } + process_lnk_output($lnk_home, &mut parent, $cmd) + } else { + start_lnk_cmd($lnk_home, $cmd); + return; + } + }}; +} + +fn process_lnk_output(lnk_home: &str, lnk_process: &mut Master, cmd: &LnkCmd) -> String { + let buf_reader = BufReader::new(lnk_process); + let mut output = String::new(); + for line in buf_reader.lines() { + let line = line.unwrap(); + + // Print the output and decode them if necessary. + println!("{}: {}", lnk_home, line); + match cmd { + LnkCmd::IdPersonCreate { name: _ } => { + if line.find("\"urn\":").is_some() { + output = line; // get the line with URN. + } + }, + LnkCmd::IdProjectCreate { name: _ } => { + if line.find("\"urn\":").is_some() { + output = line; // get the line with URN. + } + }, + LnkCmd::ProfileGet => { + output = line; // get the last line for profile id. + }, + LnkCmd::ProfilePeer => { + output = line; // get the last line for peer id. + }, + LnkCmd::Clone { + urn: _, + peer_id: _, + path: _, + } => { + output = line; + }, + _ => {}, + } + } + + output +} + +fn start_lnk_cmd(lnk_home: &str, cmd: &LnkCmd) { + let manifest_path = manifest_path(); + + // cargo run \ + // --manifest-path $LINK_CHECKOUT/bins/Cargo.toml \ + // -p lnk -- "$@" + let mut lnk_cmd = Command::new("cargo"); + lnk_cmd + .env("LNK_HOME", lnk_home) + .arg("run") + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("lnk") + .arg("--"); + let full_cmd = match cmd { + LnkCmd::ProfileCreate => lnk_cmd.arg("profile").arg("create"), + LnkCmd::ProfileGet => lnk_cmd.arg("profile").arg("get"), + LnkCmd::ProfilePeer => lnk_cmd.arg("profile").arg("peer"), + LnkCmd::ProfileSshAdd => lnk_cmd.arg("profile").arg("ssh").arg("add"), + LnkCmd::IdPersonCreate { name } => { + let payload = json!({ "name": name }); + lnk_cmd + .arg("identities") + .arg("person") + .arg("create") + .arg("new") + .arg("--payload") + .arg(payload.to_string()) + }, + LnkCmd::IdLocalSet { urn } => lnk_cmd + .arg("identities") + .arg("local") + .arg("set") + .arg("--urn") + .arg(urn), + LnkCmd::IdProjectCreate { name } => { + let payload = json!({"name": name, "default_branch": "master"}); + let project_path = format!("./{}", name); + lnk_cmd + .arg("identities") + .arg("project") + .arg("create") + .arg("new") + .arg("--path") + .arg(project_path) + .arg("--payload") + .arg(payload.to_string()) + }, + LnkCmd::Clone { urn, peer_id, path } => lnk_cmd + .arg("clone") + .arg("--urn") + .arg(urn) + .arg("--path") + .arg(path) + .arg("--peer") + .arg(peer_id), + }; + full_cmd.status().expect("lnk cmd failed:"); +} + +fn spawn_linkd(lnk_home: &str, manifest_path: &str) -> Child { + let log_name = format!("linkd_{}.log", ×tamp()); + let log_file = File::create(&log_name).unwrap(); + let target_dir = bins_target_dir(); + let exec_path = format!("{}/debug/linkd", &target_dir); + + Command::new("cargo") + .arg("build") + .arg("--target-dir") + .arg(&target_dir) + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("linkd") + .output() + .expect("cargo build linkd failed"); + + let child = Command::new(&exec_path) + .env("RUST_BACKTRACE", "1") + .arg("--lnk-home") + .arg(lnk_home) + .arg("--track") + .arg("everything") + .arg("--protocol-listen") + .arg("127.0.0.1:8799") + .stdout(Stdio::from(log_file)) + .spawn() + .expect("linkd failed to start"); + println!("linkd stdout redirected to {}", &log_name); + thread::sleep(Duration::from_secs(1)); + child +} + +fn spawn_lnk_gitd(lnk_home: &str, manifest_path: &str, addr: &str) -> Child { + let target_dir = bins_target_dir(); + let exec_path = format!("{}/debug/lnk-gitd", &target_dir); + + Command::new("cargo") + .arg("build") + .arg("--target-dir") + .arg(&target_dir) + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("lnk-gitd") + .stdout(Stdio::inherit()) + .output() + .expect("cargo build lnk-gitd failed"); + + let child = Command::new(&exec_path) + .arg(lnk_home) + .arg("--push-seeds") + .arg("--fetch-seeds") + .arg("-a") + .arg(addr) + .spawn() + .expect("lnk-gitd failed to start"); + println!("started lnk-gitd"); + child +} + +/// Returns true if runs in the forked child process for git push. +/// returns false if runs in the parent process. +fn run_git_push_in_child_process() -> bool { + let fork = Fork::from_ptmx().unwrap(); + if let Some(mut parent) = fork.is_parent().ok() { + let yes = b"yes\n"; + let buf_reader = BufReader::new(parent); + for line in buf_reader.lines() { + let line = line.unwrap(); + println!("git-push: {}", line); + if line.find("key fingerprint").is_some() { + parent.write_all(yes).unwrap(); + } + } + + false // This is not the child process. + } else { + Command::new("git") + .arg("push") + .arg("linkd") + .status() + .expect("failed to do git push"); + true // This is the child process. + } +} + +fn git_last_commit() -> String { + let output = Command::new("git") + .arg("rev-parse") + .arg("HEAD") + .output() + .expect("failed to run git rev-parse"); + String::from_utf8_lossy(&output.stdout).to_string() +} + +/// Returns UNIX_TIME in millis. +fn timestamp() -> u128 { + let now = SystemTime::now() + .duration_since(SystemTime::UNIX_EPOCH) + .unwrap(); + now.as_millis() +} + +/// Returns the full path of `bins` manifest file. +fn manifest_path() -> String { + let package_dir = env!("CARGO_MANIFEST_DIR"); + format!("{}/Cargo.toml", package_dir.strip_suffix("/tests").unwrap()) +} + +/// Returns the full path of `bins/target`. +fn bins_target_dir() -> String { + let package_dir = env!("CARGO_MANIFEST_DIR"); + format!("{}/target", package_dir.strip_suffix("/tests").unwrap()) +} + +fn clean_up_known_hosts() { + // ssh-keygen -f "/home/pi/.ssh/known_hosts" -R "[127.0.0.1]:9987" + let home_dir = env!("HOME"); + let known_hosts = format!("{}/.ssh/known_hosts", &home_dir); + let output = Command::new("ssh-keygen") + .arg("-f") + .arg(known_hosts) + .arg("-R") + .arg("[127.0.0.1]:9987") + .output() + .expect("failed to do ssh-keygen"); + println!("ssh-keygen: {:?}", &output); +} diff --git a/link-git/src/protocol/fetch.rs b/link-git/src/protocol/fetch.rs index a5a73065..98b92b51 100644 --- a/link-git/src/protocol/fetch.rs +++ b/link-git/src/protocol/fetch.rs @@ -38,8 +38,11 @@ use super::{packwriter::PackWriter, remote_git_version, transport}; // // cf. https://lore.kernel.org/git/CD2XNXHACAXS.13J6JTWZPO1JA@schmidt/ // Fixed in `git.git` 1ab13eb, which should land in 2.34 +// +// Based on testing with git 2.25.1 in Ubuntu 20.04, this workaround is +// not needed. Hence the checked version is lowered to 2.25.0. fn must_namespace_want_ref(caps: &client::Capabilities) -> bool { - static FIXED_AFTER: Lazy<Version> = Lazy::new(|| Version::new("2.33.0").unwrap()); + static FIXED_AFTER: Lazy<Version> = Lazy::new(|| Version::new("2.25.0").unwrap()); remote_git_version(caps) .map(|version| version <= *FIXED_AFTER) diff --git a/link-git/src/protocol/ls.rs b/link-git/src/protocol/ls.rs index b3516454..de95b4a3 100644 --- a/link-git/src/protocol/ls.rs +++ b/link-git/src/protocol/ls.rs @@ -22,9 +22,12 @@ use super::{remote_git_version, transport}; // Work around `git-upload-pack` not handling namespaces properly // // cf. https://lore.kernel.org/git/pMV5dJabxOBTD8kJBaPuWK0aS6OJhRQ7YFGwfhPCeSJEbPDrIFBza36nXBCgUCeUJWGmpjPI1rlOGvZJEh71Ruz4SqljndUwOCoBUDRHRDU=@eagain.st/ +// +// Based on testing with git 2.25.1 in Ubuntu 20.04, this workaround is +// not needed. Hence the checked version is lowered to 2.25.0. fn must_namespace(caps: &client::Capabilities) -> bool { static MIN_GIT_VERSION_NAMESPACES: Lazy<Version> = - Lazy::new(|| Version::new("2.31.0").unwrap()); + Lazy::new(|| Version::new("2.25.0").unwrap()); remote_git_version(caps) .map(|version| version < *MIN_GIT_VERSION_NAMESPACES) -- 2.32.0 (Apple Git-132)
--- bins/Cargo.lock | 29 +++ bins/Cargo.toml | 1 + bins/tests/Cargo.toml | 13 + bins/tests/integration_test.rs | 456 +++++++++++++++++++++++++++++++++ link-git/src/protocol/fetch.rs | 5 +- link-git/src/protocol/ls.rs | 5 +- 6 files changed, 507 insertions(+), 2 deletions(-) create mode 100644 bins/tests/Cargo.toml create mode 100644 bins/tests/integration_test.rs diff --git a/bins/Cargo.lock b/bins/Cargo.lock index ca8d4229..25d0898b 100644 --- a/bins/Cargo.lock @@ -840,6 +840,17 @@ dependencies = [ "termcolor", ] +[[package]] +name = "errno" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e2b2decb0484e15560df3210cf0d78654bb0864b2c138977c07e377a1bae0e2" +dependencies = [ + "kernel32-sys", + "libc", + "winapi 0.2.8", +] + [[package]] name = "event-listener" version = "2.5.3" @@ -2943,6 +2954,16 @@ dependencies = [ "human_format", ] +[[package]] +name = "pty" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f50f3d255966981eb4e4c5df3e983e6f7d163221f547406d83b6a460ff5c5ee8" +dependencies = [ + "errno", + "libc", +] + [[package]] name = "quanta" version = "0.4.1" @@ -3568,6 +3589,14 @@ dependencies = [ "winapi-util", ] +[[package]] +name = "tests" +version = "0.1.0" +dependencies = [ + "pty", + "serde_json", +] + [[package]] name = "textwrap" version = "0.15.0" diff --git a/bins/Cargo.toml b/bins/Cargo.toml index 3d1786f4..99b3c69f 100644 --- a/bins/Cargo.toml @@ -5,4 +5,5 @@ members = [ "lnk-gitd", "lnk-identities-dev", "lnk-profile-dev", + "tests", ] diff --git a/bins/tests/Cargo.toml b/bins/tests/Cargo.toml new file mode 100644 index 00000000..3ae82769 --- /dev/null @@ -0,0 +1,13 @@ +[package] +name = "tests" +version = "0.1.0" +authors = ["Han Xu <keepsimple@gmail.com>"] +edition = "2021" + +[dev-dependencies] +pty = "0.2" +serde_json = "1.0" + +[[test]] +name = "integration_test" +path = "integration_test.rs" diff --git a/bins/tests/integration_test.rs b/bins/tests/integration_test.rs new file mode 100644 index 00000000..2d2a4a92 --- /dev/null @@ -0,0 +1,456 @@ +// Copyright © 2022 The Radicle Link Contributors +// +// This file is part of radicle-link, distributed under the GPLv3 with Radicle +// Linking Exception. For full terms see the included LICENSE file. + +//! Integration test to exercise the programs in `bins`. + +use pty::fork::*; +use serde_json::{json, Value}; +use std::{ + env, + fs::File, + io::{BufRead, BufReader, Write}, + process::{Child, Command, Stdio}, + thread, + time::{Duration, SystemTime}, +}; + +/// This test is inspired by https://github.com/alexjg/linkd-playground +/// +/// Tests a typical scenario: there are two peer nodes and one seed node. +/// The main steps are: +/// - Setup a profile for each node in tmp home directories. +/// - Setup SSH keys for each profile. +/// - Create identities. +/// - Create a local repo for peer 1. +/// - Start `linkd` for the seed, and `lnk-gitd` for peer 1. +/// - Push peer 1 repo to its monorepo and to the seed. +/// - Clone the peer 1 repo to peer 2 via seed. +/// +/// This test only works in Linux at this moment. +#[test] +fn two_peers_and_a_seed() { + let peer1_home = "/tmp/link-local-1"; + let peer2_home = "/tmp/link-local-2"; + let seed_home = "/tmp/seed-home"; + let passphrase = b"play\n"; + + { + println!("\n== create lnk homes for two peers and one seed ==\n"); + let cmd = LnkCmd::ProfileCreate; + run_lnk!(&cmd, peer1_home, passphrase); + run_lnk!(&cmd, peer2_home, passphrase); + run_lnk!(&cmd, seed_home, passphrase); + } + + { + println!("\n== add ssh keys for each profile to the ssh-agent ==\n"); + let cmd = LnkCmd::ProfileSshAdd; + run_lnk!(&cmd, peer1_home, passphrase); + run_lnk!(&cmd, peer2_home, passphrase); + run_lnk!(&cmd, seed_home, passphrase); + } + + { + println!("\n== Creating local link 1 identity ==\n"); + let peer1_name = "sockpuppet1".to_string(); + let cmd = LnkCmd::IdPersonCreate { name: peer1_name }; + let output = run_lnk!(&cmd, peer1_home, passphrase); + let v: Value = serde_json::from_str(&output).unwrap(); + let urn = v["urn"].as_str().unwrap().to_string(); + let cmd = LnkCmd::IdLocalSet { urn }; + run_lnk!(&cmd, peer1_home, passphrase); + } + + { + println!("\n== Creating local link 2 identity ==\n"); + let peer2_name = "sockpuppet2".to_string(); + let cmd = LnkCmd::IdPersonCreate { name: peer2_name }; + let output = run_lnk!(&cmd, peer2_home, passphrase); + let v: Value = serde_json::from_str(&output).unwrap(); + let urn = v["urn"].as_str().unwrap().to_string(); + let cmd = LnkCmd::IdLocalSet { urn }; + run_lnk!(&cmd, peer2_home, passphrase); + } + + let peer1_proj_dir = format!("peer1_proj_{}", timestamp()); + let peer1_proj_urn = { + println!("\n== Create a local repository for peer1 ==\n"); + let cmd = LnkCmd::IdProjectCreate { + name: peer1_proj_dir.clone(), + }; + let output = run_lnk!(&cmd, peer1_home, passphrase); + let v: Value = serde_json::from_str(&output).unwrap(); + let proj_urn = v["urn"].as_str().unwrap().to_string(); + println!("our project URN: {}", &proj_urn); + proj_urn + }; + + println!("\n== Add the seed to the local peer seed configs ==\n"); + let seed_endpoint = { + let cmd = LnkCmd::ProfilePeer; + let seed_peer_id = run_lnk!(&cmd, seed_home, passphrase); + format!("{}@127.0.0.1:8799", &seed_peer_id) + }; + + { + // Create seed file for peer1 + let cmd = LnkCmd::ProfileGet; + let peer1_profile = run_lnk!(&cmd, peer1_home, passphrase); + let peer1_seed = format!("{}/{}/seeds", peer1_home, peer1_profile); + let mut peer1_seed_f = File::create(peer1_seed).unwrap(); + peer1_seed_f.write_all(seed_endpoint.as_bytes()).unwrap(); + } + + { + // Create seed file for peer2 + let cmd = LnkCmd::ProfileGet; + let peer2_profile = run_lnk!(&cmd, peer2_home, passphrase); + let peer2_seed = format!("{}/{}/seeds", peer2_home, peer2_profile); + let mut peer2_seed_f = File::create(peer2_seed).unwrap(); + peer2_seed_f.write_all(seed_endpoint.as_bytes()).unwrap(); + } + + println!("\n== Start the seed linkd ==\n"); + let manifest_path = manifest_path(); + let mut linkd = spawn_linkd(seed_home, &manifest_path); + + println!("\n== Start the peer 1 gitd ==\n"); + let cmd = LnkCmd::ProfilePeer; + let gitd_addr = "127.0.0.1:9987"; + let peer1_peer_id = run_lnk!(&cmd, peer1_home, passphrase); + let mut gitd = spawn_lnk_gitd(peer1_home, &manifest_path, gitd_addr); + + println!("\n== Make some changes in the repo: add and commit a test file ==\n"); + env::set_current_dir(&peer1_proj_dir).unwrap(); + let mut test_file = File::create("test").unwrap(); + test_file.write_all(b"test").unwrap(); + Command::new("git") + .arg("add") + .arg("test") + .output() + .expect("failed to do git add"); + let output = Command::new("git") + .arg("commit") + .arg("-m") + .arg("test commit") + .output() + .expect("failed to do git commit"); + println!("git-commit: {:?}", &output); + + println!("\n== Add the linkd remote to the repo ==\n"); + let remote_url = format!("ssh://rad@{}/{}.git", gitd_addr, &peer1_proj_urn); + Command::new("git") + .arg("remote") + .arg("add") + .arg("linkd") + .arg(remote_url) + .output() + .expect("failed to do git remote add"); + + clean_up_known_hosts(); + + if run_git_push_in_child_process() { + // The child process is done with git push. + return; + } + + let peer1_last_commit = git_last_commit(); + + println!("\n== Clone to peer2 ==\n"); + + env::set_current_dir("..").unwrap(); // out of the peer1 proj directory. + let peer2_proj = format!("peer2_proj_{}", timestamp()); + let cmd = LnkCmd::Clone { + urn: peer1_proj_urn, + peer_id: peer1_peer_id, + path: peer2_proj.clone(), + }; + run_lnk!(&cmd, peer2_home, passphrase); + + env::set_current_dir(peer2_proj).unwrap(); + let peer2_last_commit = git_last_commit(); + println!("\n== peer1 proj last commit: {}", &peer1_last_commit); + println!("\n== peer2 proj last commit: {}", &peer2_last_commit); + + println!("\n== Cleanup: kill linkd (seed) and gitd (peer1) ==\n"); + + linkd.kill().ok(); + gitd.kill().ok(); + + assert_eq!(peer1_last_commit, peer2_last_commit); +} + +enum LnkCmd { + ProfileCreate, + ProfileGet, + ProfilePeer, + ProfileSshAdd, + IdPersonCreate { + name: String, + }, + IdLocalSet { + urn: String, + }, + IdProjectCreate { + name: String, + }, + Clone { + urn: String, + peer_id: String, + path: String, + }, +} + +/// Runs a `lnk` command of `$cmd` using `$lnk_home` as the node home. +/// Also support the passphrase input for commands that need it. +#[macro_export] +macro_rules! run_lnk { + ( $cmd:expr, $lnk_home:ident, $passphrase:ident ) => {{ + let fork = Fork::from_ptmx().unwrap(); + if let Some(mut parent) = fork.is_parent().ok() { + match $cmd { + LnkCmd::ProfileCreate | LnkCmd::ProfileSshAdd => { + // Input the passphrase if necessary. + parent.write_all($passphrase).unwrap(); + }, + _ => {}, + } + process_lnk_output($lnk_home, &mut parent, $cmd) + } else { + start_lnk_cmd($lnk_home, $cmd); + return; + } + }}; +} + +fn process_lnk_output(lnk_home: &str, lnk_process: &mut Master, cmd: &LnkCmd) -> String { + let buf_reader = BufReader::new(lnk_process); + let mut output = String::new(); + for line in buf_reader.lines() { + let line = line.unwrap(); + + // Print the output and decode them if necessary. + println!("{}: {}", lnk_home, line); + match cmd { + LnkCmd::IdPersonCreate { name: _ } => { + if line.find("\"urn\":").is_some() { + output = line; // get the line with URN. + } + }, + LnkCmd::IdProjectCreate { name: _ } => { + if line.find("\"urn\":").is_some() { + output = line; // get the line with URN. + } + }, + LnkCmd::ProfileGet => { + output = line; // get the last line for profile id. + }, + LnkCmd::ProfilePeer => { + output = line; // get the last line for peer id. + }, + LnkCmd::Clone { + urn: _, + peer_id: _, + path: _, + } => { + output = line; + }, + _ => {}, + } + } + + output +} + +fn start_lnk_cmd(lnk_home: &str, cmd: &LnkCmd) { + let manifest_path = manifest_path(); + + // cargo run \ + // --manifest-path $LINK_CHECKOUT/bins/Cargo.toml \ + // -p lnk -- "$@" + let mut lnk_cmd = Command::new("cargo"); + lnk_cmd + .env("LNK_HOME", lnk_home) + .arg("run") + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("lnk") + .arg("--"); + let full_cmd = match cmd { + LnkCmd::ProfileCreate => lnk_cmd.arg("profile").arg("create"), + LnkCmd::ProfileGet => lnk_cmd.arg("profile").arg("get"), + LnkCmd::ProfilePeer => lnk_cmd.arg("profile").arg("peer"), + LnkCmd::ProfileSshAdd => lnk_cmd.arg("profile").arg("ssh").arg("add"), + LnkCmd::IdPersonCreate { name } => { + let payload = json!({ "name": name }); + lnk_cmd + .arg("identities") + .arg("person") + .arg("create") + .arg("new") + .arg("--payload") + .arg(payload.to_string()) + }, + LnkCmd::IdLocalSet { urn } => lnk_cmd + .arg("identities") + .arg("local") + .arg("set") + .arg("--urn") + .arg(urn), + LnkCmd::IdProjectCreate { name } => { + let payload = json!({"name": name, "default_branch": "master"}); + let project_path = format!("./{}", name); + lnk_cmd + .arg("identities") + .arg("project") + .arg("create") + .arg("new") + .arg("--path") + .arg(project_path) + .arg("--payload") + .arg(payload.to_string()) + }, + LnkCmd::Clone { urn, peer_id, path } => lnk_cmd + .arg("clone") + .arg("--urn") + .arg(urn) + .arg("--path") + .arg(path) + .arg("--peer") + .arg(peer_id), + }; + full_cmd.status().expect("lnk cmd failed:"); +} + +fn spawn_linkd(lnk_home: &str, manifest_path: &str) -> Child { + let log_name = format!("linkd_{}.log", ×tamp()); + let log_file = File::create(&log_name).unwrap(); + let target_dir = bins_target_dir(); + let exec_path = format!("{}/debug/linkd", &target_dir); + + Command::new("cargo") + .arg("build") + .arg("--target-dir") + .arg(&target_dir) + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("linkd") + .output() + .expect("cargo build linkd failed"); + + let child = Command::new(&exec_path) + .env("RUST_BACKTRACE", "1") + .arg("--lnk-home") + .arg(lnk_home) + .arg("--track") + .arg("everything") + .arg("--protocol-listen") + .arg("127.0.0.1:8799") + .stdout(Stdio::from(log_file)) + .spawn() + .expect("linkd failed to start"); + println!("linkd stdout redirected to {}", &log_name); + thread::sleep(Duration::from_secs(1)); + child +} + +fn spawn_lnk_gitd(lnk_home: &str, manifest_path: &str, addr: &str) -> Child { + let target_dir = bins_target_dir(); + let exec_path = format!("{}/debug/lnk-gitd", &target_dir); + + Command::new("cargo") + .arg("build") + .arg("--target-dir") + .arg(&target_dir) + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("lnk-gitd") + .stdout(Stdio::inherit()) + .output() + .expect("cargo build lnk-gitd failed"); + + let child = Command::new(&exec_path) + .arg(lnk_home) + .arg("--push-seeds") + .arg("--fetch-seeds") + .arg("-a") + .arg(addr) + .spawn() + .expect("lnk-gitd failed to start"); + println!("started lnk-gitd"); + child +} + +/// Returns true if runs in the forked child process for git push. +/// returns false if runs in the parent process. +fn run_git_push_in_child_process() -> bool { + let fork = Fork::from_ptmx().unwrap(); + if let Some(mut parent) = fork.is_parent().ok() { + let yes = b"yes\n"; + let buf_reader = BufReader::new(parent); + for line in buf_reader.lines() { + let line = line.unwrap(); + println!("git-push: {}", line); + if line.find("key fingerprint").is_some() { + parent.write_all(yes).unwrap(); + } + } + + false // This is not the child process. + } else { + Command::new("git") + .arg("push") + .arg("linkd") + .status() + .expect("failed to do git push"); + true // This is the child process. + } +} + +fn git_last_commit() -> String { + let output = Command::new("git") + .arg("rev-parse") + .arg("HEAD") + .output() + .expect("failed to run git rev-parse"); + String::from_utf8_lossy(&output.stdout).to_string() +} + +/// Returns UNIX_TIME in millis. +fn timestamp() -> u128 { + let now = SystemTime::now() + .duration_since(SystemTime::UNIX_EPOCH) + .unwrap(); + now.as_millis() +} + +/// Returns the full path of `bins` manifest file. +fn manifest_path() -> String { + let package_dir = env!("CARGO_MANIFEST_DIR"); + format!("{}/Cargo.toml", package_dir.strip_suffix("/tests").unwrap()) +} + +/// Returns the full path of `bins/target`. +fn bins_target_dir() -> String { + let package_dir = env!("CARGO_MANIFEST_DIR"); + format!("{}/target", package_dir.strip_suffix("/tests").unwrap()) +} + +fn clean_up_known_hosts() { + // ssh-keygen -f "/home/pi/.ssh/known_hosts" -R "[127.0.0.1]:9987" + let home_dir = env!("HOME"); + let known_hosts = format!("{}/.ssh/known_hosts", &home_dir); + let output = Command::new("ssh-keygen") + .arg("-f") + .arg(known_hosts) + .arg("-R") + .arg("[127.0.0.1]:9987") + .output() + .expect("failed to do ssh-keygen"); + println!("ssh-keygen: {:?}", &output); +} diff --git a/link-git/src/protocol/fetch.rs b/link-git/src/protocol/fetch.rs index a5a73065..98b92b51 100644 --- a/link-git/src/protocol/fetch.rs +++ b/link-git/src/protocol/fetch.rs @@ -38,8 +38,11 @@ use super::{packwriter::PackWriter, remote_git_version, transport}; // // cf. https://lore.kernel.org/git/CD2XNXHACAXS.13J6JTWZPO1JA@schmidt/ // Fixed in `git.git` 1ab13eb, which should land in 2.34 +// +// Based on testing with git 2.25.1 in Ubuntu 20.04, this workaround is +// not needed. Hence the checked version is lowered to 2.25.0. fn must_namespace_want_ref(caps: &client::Capabilities) -> bool { - static FIXED_AFTER: Lazy<Version> = Lazy::new(|| Version::new("2.33.0").unwrap()); + static FIXED_AFTER: Lazy<Version> = Lazy::new(|| Version::new("2.25.0").unwrap()); remote_git_version(caps) .map(|version| version <= *FIXED_AFTER) diff --git a/link-git/src/protocol/ls.rs b/link-git/src/protocol/ls.rs index b3516454..de95b4a3 100644 --- a/link-git/src/protocol/ls.rs +++ b/link-git/src/protocol/ls.rs @@ -22,9 +22,12 @@ use super::{remote_git_version, transport}; // Work around `git-upload-pack` not handling namespaces properly // // cf. https://lore.kernel.org/git/pMV5dJabxOBTD8kJBaPuWK0aS6OJhRQ7YFGwfhPCeSJEbPDrIFBza36nXBCgUCeUJWGmpjPI1rlOGvZJEh71Ruz4SqljndUwOCoBUDRHRDU=@eagain.st/ +// +// Based on testing with git 2.25.1 in Ubuntu 20.04, this workaround is +// not needed. Hence the checked version is lowered to 2.25.0. fn must_namespace(caps: &client::Capabilities) -> bool { static MIN_GIT_VERSION_NAMESPACES: Lazy<Version> = - Lazy::new(|| Version::new("2.31.0").unwrap()); + Lazy::new(|| Version::new("2.25.0").unwrap()); remote_git_version(caps) .map(|version| version < *MIN_GIT_VERSION_NAMESPACES) -- 2.32.0 (Apple Git-132)
--- cli/gitd-lib/src/hooks.rs | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/cli/gitd-lib/src/hooks.rs b/cli/gitd-lib/src/hooks.rs index 5e928ffd..cf37d71f 100644 --- a/cli/gitd-lib/src/hooks.rs +++ b/cli/gitd-lib/src/hooks.rs @@ -77,6 +77,16 @@ where E: std::error::Error + Send + 'static, P: ProgressReporter<Error = E>, { + // update the signed refs before a possible request_pull, + // so that the peer can receive the latest refs. + let at = update_signed_refs( + reporter, + self.spawner.clone(), + self.pool.clone(), + urn.clone(), + ) + .await?; + if self.post_receive.request_pull { tracing::info!("executing request-pull"); request_pull(reporter, &self.client, &self.seeds, urn.clone()).await?; @@ -87,14 +97,7 @@ where ) .await?; } - let at = match update_signed_refs( - reporter, - self.spawner.clone(), - self.pool.clone(), - urn.clone(), - ) - .await? - { + let at = match at { Some(at) => at, None => return Ok(()), }; -- 2.32.0 (Apple Git-132)
Signed-off-by: Han Xu <keepsimple@gmail.com> --- cli/gitd-lib/src/hooks.rs | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/cli/gitd-lib/src/hooks.rs b/cli/gitd-lib/src/hooks.rs index 5e928ffd..cf37d71f 100644 --- a/cli/gitd-lib/src/hooks.rs +++ b/cli/gitd-lib/src/hooks.rs @@ -77,6 +77,16 @@ where E: std::error::Error + Send + 'static, P: ProgressReporter<Error = E>, { + // update the signed refs before a possible request_pull, + // so that the peer can receive the latest refs. + let at = update_signed_refs( + reporter, + self.spawner.clone(), + self.pool.clone(), + urn.clone(), + ) + .await?; + if self.post_receive.request_pull { tracing::info!("executing request-pull"); request_pull(reporter, &self.client, &self.seeds, urn.clone()).await?; @@ -87,14 +97,7 @@ where ) .await?; } - let at = match update_signed_refs( - reporter, - self.spawner.clone(), - self.pool.clone(), - urn.clone(), - ) - .await? - { + let at = match at { Some(at) => at, None => return Ok(()), }; -- 2.32.1 (Apple Git-133)
--- bins/tests/integration_test.rs | 31 +++++++++++++++++++++++++------ 1 file changed, 25 insertions(+), 6 deletions(-) diff --git a/bins/tests/integration_test.rs b/bins/tests/integration_test.rs index 770f1b5e..ea6815ab 100644 --- a/bins/tests/integration_test.rs @@ -170,11 +170,15 @@ fn two_peers_and_a_seed() { return; } + let peer1_last_commit = git_last_commit(); + println!("\n== peer1 project last commit: {} ==\n", &peer1_last_commit); + println!("\n== Clone to peer2 ==\n"); env::set_current_dir("..").unwrap(); // out of the peer1 proj directory. + let peer2_proj = format!("peer2_proj_{}", timestamp()); let (is_parent, _) = run_lnk( - LnkCmd::Clone(proj_urn, peer1_peer_id), + LnkCmd::Clone(proj_urn, peer1_peer_id, peer2_proj.clone()), peer2_home, passphrase, ); @@ -182,9 +186,16 @@ fn two_peers_and_a_seed() { return; } - println!("\n== Kill linkd (seed) ==\n"); + env::set_current_dir(peer2_proj).unwrap(); + let peer2_last_commit = git_last_commit(); + println!("\n== peer1 proj last commit: {}", &peer1_last_commit); + println!("\n== peer2 proj last commit: {}", &peer2_last_commit); + + println!("\n== Cleanup: kill linkd (seed) ==\n"); linkd.kill().ok(); + + assert_eq!(peer1_last_commit, peer2_last_commit); } enum LnkCmd { @@ -195,7 +206,7 @@ enum LnkCmd { IdPersonCreate(String), // the associated string is "the person's name". IdLocalSet(String), // the associated string is "urn". IdProjectCreate(String), // the associated string is "the project name". - Clone(String, String), // the associated string is "urn", "peer_id" + Clone(String, String, String), // the associated string is "urn", "peer_id", "path" } /// Runs a `cmd` for `lnk_home`. Rebuilds `lnk` if necessary. @@ -238,7 +249,7 @@ fn run_lnk(cmd: LnkCmd, lnk_home: &str, passphrase: &[u8]) -> (bool, String) { LnkCmd::ProfilePeer => { output = line; // get the last line for peer id. }, - LnkCmd::Clone(ref _urn, ref _peer) => { + LnkCmd::Clone(ref _urn, ref _peer, ref _path) => { output = line; }, _ => {}, @@ -296,8 +307,7 @@ fn run_lnk(cmd: LnkCmd, lnk_home: &str, passphrase: &[u8]) -> (bool, String) { .arg("--payload") .arg(payload.to_string()) }, - LnkCmd::Clone(urn, peer_id) => { - let peer2_proj = format!("peer2_proj_{}", timestamp()); + LnkCmd::Clone(urn, peer_id, peer2_proj) => { lnk_cmd .arg("clone") .arg("--urn") @@ -414,6 +424,15 @@ fn run_git_push() -> bool { } } +fn git_last_commit() -> String { + let output = Command::new("git") + .arg("rev-parse") + .arg("HEAD") + .output() + .expect("failed to run git rev-parse"); + String::from_utf8_lossy(&output.stdout).to_string() +} + /// Returns UNIX_TIME in millis. fn timestamp() -> u128 { let now = SystemTime::now() -- 2.32.0 (Apple Git-132)
--- bins/tests/integration_test.rs | 31 +++++++++++++++++++++++++------ 1 file changed, 25 insertions(+), 6 deletions(-) diff --git a/bins/tests/integration_test.rs b/bins/tests/integration_test.rs index 770f1b5e..ea6815ab 100644 --- a/bins/tests/integration_test.rs @@ -170,11 +170,15 @@ fn two_peers_and_a_seed() { return; } + let peer1_last_commit = git_last_commit(); + println!("\n== peer1 project last commit: {} ==\n", &peer1_last_commit); + println!("\n== Clone to peer2 ==\n"); env::set_current_dir("..").unwrap(); // out of the peer1 proj directory. + let peer2_proj = format!("peer2_proj_{}", timestamp()); let (is_parent, _) = run_lnk( - LnkCmd::Clone(proj_urn, peer1_peer_id), + LnkCmd::Clone(proj_urn, peer1_peer_id, peer2_proj.clone()), peer2_home, passphrase, ); @@ -182,9 +186,16 @@ fn two_peers_and_a_seed() { return; } - println!("\n== Kill linkd (seed) ==\n"); + env::set_current_dir(peer2_proj).unwrap(); + let peer2_last_commit = git_last_commit(); + println!("\n== peer1 proj last commit: {}", &peer1_last_commit); + println!("\n== peer2 proj last commit: {}", &peer2_last_commit); + + println!("\n== Cleanup: kill linkd (seed) ==\n"); linkd.kill().ok(); + + assert_eq!(peer1_last_commit, peer2_last_commit); } enum LnkCmd { @@ -195,7 +206,7 @@ enum LnkCmd { IdPersonCreate(String), // the associated string is "the person's name". IdLocalSet(String), // the associated string is "urn". IdProjectCreate(String), // the associated string is "the project name". - Clone(String, String), // the associated string is "urn", "peer_id" + Clone(String, String, String), // the associated string is "urn", "peer_id", "path" } /// Runs a `cmd` for `lnk_home`. Rebuilds `lnk` if necessary. @@ -238,7 +249,7 @@ fn run_lnk(cmd: LnkCmd, lnk_home: &str, passphrase: &[u8]) -> (bool, String) { LnkCmd::ProfilePeer => { output = line; // get the last line for peer id. }, - LnkCmd::Clone(ref _urn, ref _peer) => { + LnkCmd::Clone(ref _urn, ref _peer, ref _path) => { output = line; }, _ => {}, @@ -296,8 +307,7 @@ fn run_lnk(cmd: LnkCmd, lnk_home: &str, passphrase: &[u8]) -> (bool, String) { .arg("--payload") .arg(payload.to_string()) }, - LnkCmd::Clone(urn, peer_id) => { - let peer2_proj = format!("peer2_proj_{}", timestamp()); + LnkCmd::Clone(urn, peer_id, peer2_proj) => { lnk_cmd .arg("clone") .arg("--urn") @@ -414,6 +424,15 @@ fn run_git_push() -> bool { } } +fn git_last_commit() -> String { + let output = Command::new("git") + .arg("rev-parse") + .arg("HEAD") + .output() + .expect("failed to run git rev-parse"); + String::from_utf8_lossy(&output.stdout).to_string() +} + /// Returns UNIX_TIME in millis. fn timestamp() -> u128 { let now = SystemTime::now() -- 2.32.0 (Apple Git-132)
--- cli/gitd-lib/src/hooks.rs | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/cli/gitd-lib/src/hooks.rs b/cli/gitd-lib/src/hooks.rs index 5e928ffd..cf37d71f 100644 --- a/cli/gitd-lib/src/hooks.rs +++ b/cli/gitd-lib/src/hooks.rs @@ -77,6 +77,16 @@ where E: std::error::Error + Send + 'static, P: ProgressReporter<Error = E>, { + // update the signed refs before a possible request_pull, + // so that the peer can receive the latest refs. + let at = update_signed_refs( + reporter, + self.spawner.clone(), + self.pool.clone(), + urn.clone(), + ) + .await?; + if self.post_receive.request_pull { tracing::info!("executing request-pull"); request_pull(reporter, &self.client, &self.seeds, urn.clone()).await?; @@ -87,14 +97,7 @@ where ) .await?; } - let at = match update_signed_refs( - reporter, - self.spawner.clone(), - self.pool.clone(), - urn.clone(), - ) - .await? - { + let at = match at { Some(at) => at, None => return Ok(()), }; -- 2.32.0 (Apple Git-132)
--- cli/gitd-lib/src/hooks.rs | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/cli/gitd-lib/src/hooks.rs b/cli/gitd-lib/src/hooks.rs index 5e928ffd..cf37d71f 100644 --- a/cli/gitd-lib/src/hooks.rs +++ b/cli/gitd-lib/src/hooks.rs @@ -77,6 +77,16 @@ where E: std::error::Error + Send + 'static, P: ProgressReporter<Error = E>, { + // update the signed refs before a possible request_pull, + // so that the peer can receive the latest refs. + let at = update_signed_refs( + reporter, + self.spawner.clone(), + self.pool.clone(), + urn.clone(), + ) + .await?; + if self.post_receive.request_pull { tracing::info!("executing request-pull"); request_pull(reporter, &self.client, &self.seeds, urn.clone()).await?; @@ -87,14 +97,7 @@ where ) .await?; } - let at = match update_signed_refs( - reporter, - self.spawner.clone(), - self.pool.clone(), - urn.clone(), - ) - .await? - { + let at = match at { Some(at) => at, None => return Ok(()), }; -- 2.32.0 (Apple Git-132)
--- bins/Cargo.lock | 29 +++ bins/Cargo.toml | 1 + bins/tests/Cargo.toml | 13 + bins/tests/integration_test.rs | 456 +++++++++++++++++++++++++++++++++ 4 files changed, 499 insertions(+) create mode 100644 bins/tests/Cargo.toml create mode 100644 bins/tests/integration_test.rs diff --git a/bins/Cargo.lock b/bins/Cargo.lock index ca8d4229..25d0898b 100644 --- a/bins/Cargo.lock @@ -840,6 +840,17 @@ dependencies = [ "termcolor", ] +[[package]] +name = "errno" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e2b2decb0484e15560df3210cf0d78654bb0864b2c138977c07e377a1bae0e2" +dependencies = [ + "kernel32-sys", + "libc", + "winapi 0.2.8", +] + [[package]] name = "event-listener" version = "2.5.3" @@ -2943,6 +2954,16 @@ dependencies = [ "human_format", ] +[[package]] +name = "pty" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f50f3d255966981eb4e4c5df3e983e6f7d163221f547406d83b6a460ff5c5ee8" +dependencies = [ + "errno", + "libc", +] + [[package]] name = "quanta" version = "0.4.1" @@ -3568,6 +3589,14 @@ dependencies = [ "winapi-util", ] +[[package]] +name = "tests" +version = "0.1.0" +dependencies = [ + "pty", + "serde_json", +] + [[package]] name = "textwrap" version = "0.15.0" diff --git a/bins/Cargo.toml b/bins/Cargo.toml index 3d1786f4..99b3c69f 100644 --- a/bins/Cargo.toml @@ -5,4 +5,5 @@ members = [ "lnk-gitd", "lnk-identities-dev", "lnk-profile-dev", + "tests", ] diff --git a/bins/tests/Cargo.toml b/bins/tests/Cargo.toml new file mode 100644 index 00000000..3ae82769 --- /dev/null @@ -0,0 +1,13 @@ +[package] +name = "tests" +version = "0.1.0" +authors = ["Han Xu <keepsimple@gmail.com>"] +edition = "2021" + +[dev-dependencies] +pty = "0.2" +serde_json = "1.0" + +[[test]] +name = "integration_test" +path = "integration_test.rs" diff --git a/bins/tests/integration_test.rs b/bins/tests/integration_test.rs new file mode 100644 index 00000000..2d2a4a92 --- /dev/null @@ -0,0 +1,456 @@ +// Copyright © 2022 The Radicle Link Contributors +// +// This file is part of radicle-link, distributed under the GPLv3 with Radicle +// Linking Exception. For full terms see the included LICENSE file. + +//! Integration test to exercise the programs in `bins`. + +use pty::fork::*; +use serde_json::{json, Value}; +use std::{ + env, + fs::File, + io::{BufRead, BufReader, Write}, + process::{Child, Command, Stdio}, + thread, + time::{Duration, SystemTime}, +}; + +/// This test is inspired by https://github.com/alexjg/linkd-playground +/// +/// Tests a typical scenario: there are two peer nodes and one seed node. +/// The main steps are: +/// - Setup a profile for each node in tmp home directories. +/// - Setup SSH keys for each profile. +/// - Create identities. +/// - Create a local repo for peer 1. +/// - Start `linkd` for the seed, and `lnk-gitd` for peer 1. +/// - Push peer 1 repo to its monorepo and to the seed. +/// - Clone the peer 1 repo to peer 2 via seed. +/// +/// This test only works in Linux at this moment. +#[test] +fn two_peers_and_a_seed() { + let peer1_home = "/tmp/link-local-1"; + let peer2_home = "/tmp/link-local-2"; + let seed_home = "/tmp/seed-home"; + let passphrase = b"play\n"; + + { + println!("\n== create lnk homes for two peers and one seed ==\n"); + let cmd = LnkCmd::ProfileCreate; + run_lnk!(&cmd, peer1_home, passphrase); + run_lnk!(&cmd, peer2_home, passphrase); + run_lnk!(&cmd, seed_home, passphrase); + } + + { + println!("\n== add ssh keys for each profile to the ssh-agent ==\n"); + let cmd = LnkCmd::ProfileSshAdd; + run_lnk!(&cmd, peer1_home, passphrase); + run_lnk!(&cmd, peer2_home, passphrase); + run_lnk!(&cmd, seed_home, passphrase); + } + + { + println!("\n== Creating local link 1 identity ==\n"); + let peer1_name = "sockpuppet1".to_string(); + let cmd = LnkCmd::IdPersonCreate { name: peer1_name }; + let output = run_lnk!(&cmd, peer1_home, passphrase); + let v: Value = serde_json::from_str(&output).unwrap(); + let urn = v["urn"].as_str().unwrap().to_string(); + let cmd = LnkCmd::IdLocalSet { urn }; + run_lnk!(&cmd, peer1_home, passphrase); + } + + { + println!("\n== Creating local link 2 identity ==\n"); + let peer2_name = "sockpuppet2".to_string(); + let cmd = LnkCmd::IdPersonCreate { name: peer2_name }; + let output = run_lnk!(&cmd, peer2_home, passphrase); + let v: Value = serde_json::from_str(&output).unwrap(); + let urn = v["urn"].as_str().unwrap().to_string(); + let cmd = LnkCmd::IdLocalSet { urn }; + run_lnk!(&cmd, peer2_home, passphrase); + } + + let peer1_proj_dir = format!("peer1_proj_{}", timestamp()); + let peer1_proj_urn = { + println!("\n== Create a local repository for peer1 ==\n"); + let cmd = LnkCmd::IdProjectCreate { + name: peer1_proj_dir.clone(), + }; + let output = run_lnk!(&cmd, peer1_home, passphrase); + let v: Value = serde_json::from_str(&output).unwrap(); + let proj_urn = v["urn"].as_str().unwrap().to_string(); + println!("our project URN: {}", &proj_urn); + proj_urn + }; + + println!("\n== Add the seed to the local peer seed configs ==\n"); + let seed_endpoint = { + let cmd = LnkCmd::ProfilePeer; + let seed_peer_id = run_lnk!(&cmd, seed_home, passphrase); + format!("{}@127.0.0.1:8799", &seed_peer_id) + }; + + { + // Create seed file for peer1 + let cmd = LnkCmd::ProfileGet; + let peer1_profile = run_lnk!(&cmd, peer1_home, passphrase); + let peer1_seed = format!("{}/{}/seeds", peer1_home, peer1_profile); + let mut peer1_seed_f = File::create(peer1_seed).unwrap(); + peer1_seed_f.write_all(seed_endpoint.as_bytes()).unwrap(); + } + + { + // Create seed file for peer2 + let cmd = LnkCmd::ProfileGet; + let peer2_profile = run_lnk!(&cmd, peer2_home, passphrase); + let peer2_seed = format!("{}/{}/seeds", peer2_home, peer2_profile); + let mut peer2_seed_f = File::create(peer2_seed).unwrap(); + peer2_seed_f.write_all(seed_endpoint.as_bytes()).unwrap(); + } + + println!("\n== Start the seed linkd ==\n"); + let manifest_path = manifest_path(); + let mut linkd = spawn_linkd(seed_home, &manifest_path); + + println!("\n== Start the peer 1 gitd ==\n"); + let cmd = LnkCmd::ProfilePeer; + let gitd_addr = "127.0.0.1:9987"; + let peer1_peer_id = run_lnk!(&cmd, peer1_home, passphrase); + let mut gitd = spawn_lnk_gitd(peer1_home, &manifest_path, gitd_addr); + + println!("\n== Make some changes in the repo: add and commit a test file ==\n"); + env::set_current_dir(&peer1_proj_dir).unwrap(); + let mut test_file = File::create("test").unwrap(); + test_file.write_all(b"test").unwrap(); + Command::new("git") + .arg("add") + .arg("test") + .output() + .expect("failed to do git add"); + let output = Command::new("git") + .arg("commit") + .arg("-m") + .arg("test commit") + .output() + .expect("failed to do git commit"); + println!("git-commit: {:?}", &output); + + println!("\n== Add the linkd remote to the repo ==\n"); + let remote_url = format!("ssh://rad@{}/{}.git", gitd_addr, &peer1_proj_urn); + Command::new("git") + .arg("remote") + .arg("add") + .arg("linkd") + .arg(remote_url) + .output() + .expect("failed to do git remote add"); + + clean_up_known_hosts(); + + if run_git_push_in_child_process() { + // The child process is done with git push. + return; + } + + let peer1_last_commit = git_last_commit(); + + println!("\n== Clone to peer2 ==\n"); + + env::set_current_dir("..").unwrap(); // out of the peer1 proj directory. + let peer2_proj = format!("peer2_proj_{}", timestamp()); + let cmd = LnkCmd::Clone { + urn: peer1_proj_urn, + peer_id: peer1_peer_id, + path: peer2_proj.clone(), + }; + run_lnk!(&cmd, peer2_home, passphrase); + + env::set_current_dir(peer2_proj).unwrap(); + let peer2_last_commit = git_last_commit(); + println!("\n== peer1 proj last commit: {}", &peer1_last_commit); + println!("\n== peer2 proj last commit: {}", &peer2_last_commit); + + println!("\n== Cleanup: kill linkd (seed) and gitd (peer1) ==\n"); + + linkd.kill().ok(); + gitd.kill().ok(); + + assert_eq!(peer1_last_commit, peer2_last_commit); +} + +enum LnkCmd { + ProfileCreate, + ProfileGet, + ProfilePeer, + ProfileSshAdd, + IdPersonCreate { + name: String, + }, + IdLocalSet { + urn: String, + }, + IdProjectCreate { + name: String, + }, + Clone { + urn: String, + peer_id: String, + path: String, + }, +} + +/// Runs a `lnk` command of `$cmd` using `$lnk_home` as the node home. +/// Also support the passphrase input for commands that need it. +#[macro_export] +macro_rules! run_lnk { + ( $cmd:expr, $lnk_home:ident, $passphrase:ident ) => {{ + let fork = Fork::from_ptmx().unwrap(); + if let Some(mut parent) = fork.is_parent().ok() { + match $cmd { + LnkCmd::ProfileCreate | LnkCmd::ProfileSshAdd => { + // Input the passphrase if necessary. + parent.write_all($passphrase).unwrap(); + }, + _ => {}, + } + process_lnk_output($lnk_home, &mut parent, $cmd) + } else { + start_lnk_cmd($lnk_home, $cmd); + return; + } + }}; +} + +fn process_lnk_output(lnk_home: &str, lnk_process: &mut Master, cmd: &LnkCmd) -> String { + let buf_reader = BufReader::new(lnk_process); + let mut output = String::new(); + for line in buf_reader.lines() { + let line = line.unwrap(); + + // Print the output and decode them if necessary. + println!("{}: {}", lnk_home, line); + match cmd { + LnkCmd::IdPersonCreate { name: _ } => { + if line.find("\"urn\":").is_some() { + output = line; // get the line with URN. + } + }, + LnkCmd::IdProjectCreate { name: _ } => { + if line.find("\"urn\":").is_some() { + output = line; // get the line with URN. + } + }, + LnkCmd::ProfileGet => { + output = line; // get the last line for profile id. + }, + LnkCmd::ProfilePeer => { + output = line; // get the last line for peer id. + }, + LnkCmd::Clone { + urn: _, + peer_id: _, + path: _, + } => { + output = line; + }, + _ => {}, + } + } + + output +} + +fn start_lnk_cmd(lnk_home: &str, cmd: &LnkCmd) { + let manifest_path = manifest_path(); + + // cargo run \ + // --manifest-path $LINK_CHECKOUT/bins/Cargo.toml \ + // -p lnk -- "$@" + let mut lnk_cmd = Command::new("cargo"); + lnk_cmd + .env("LNK_HOME", lnk_home) + .arg("run") + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("lnk") + .arg("--"); + let full_cmd = match cmd { + LnkCmd::ProfileCreate => lnk_cmd.arg("profile").arg("create"), + LnkCmd::ProfileGet => lnk_cmd.arg("profile").arg("get"), + LnkCmd::ProfilePeer => lnk_cmd.arg("profile").arg("peer"), + LnkCmd::ProfileSshAdd => lnk_cmd.arg("profile").arg("ssh").arg("add"), + LnkCmd::IdPersonCreate { name } => { + let payload = json!({ "name": name }); + lnk_cmd + .arg("identities") + .arg("person") + .arg("create") + .arg("new") + .arg("--payload") + .arg(payload.to_string()) + }, + LnkCmd::IdLocalSet { urn } => lnk_cmd + .arg("identities") + .arg("local") + .arg("set") + .arg("--urn") + .arg(urn), + LnkCmd::IdProjectCreate { name } => { + let payload = json!({"name": name, "default_branch": "master"}); + let project_path = format!("./{}", name); + lnk_cmd + .arg("identities") + .arg("project") + .arg("create") + .arg("new") + .arg("--path") + .arg(project_path) + .arg("--payload") + .arg(payload.to_string()) + }, + LnkCmd::Clone { urn, peer_id, path } => lnk_cmd + .arg("clone") + .arg("--urn") + .arg(urn) + .arg("--path") + .arg(path) + .arg("--peer") + .arg(peer_id), + }; + full_cmd.status().expect("lnk cmd failed:"); +} + +fn spawn_linkd(lnk_home: &str, manifest_path: &str) -> Child { + let log_name = format!("linkd_{}.log", ×tamp()); + let log_file = File::create(&log_name).unwrap(); + let target_dir = bins_target_dir(); + let exec_path = format!("{}/debug/linkd", &target_dir); + + Command::new("cargo") + .arg("build") + .arg("--target-dir") + .arg(&target_dir) + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("linkd") + .output() + .expect("cargo build linkd failed"); + + let child = Command::new(&exec_path) + .env("RUST_BACKTRACE", "1") + .arg("--lnk-home") + .arg(lnk_home) + .arg("--track") + .arg("everything") + .arg("--protocol-listen") + .arg("127.0.0.1:8799") + .stdout(Stdio::from(log_file)) + .spawn() + .expect("linkd failed to start"); + println!("linkd stdout redirected to {}", &log_name); + thread::sleep(Duration::from_secs(1)); + child +} + +fn spawn_lnk_gitd(lnk_home: &str, manifest_path: &str, addr: &str) -> Child { + let target_dir = bins_target_dir(); + let exec_path = format!("{}/debug/lnk-gitd", &target_dir); + + Command::new("cargo") + .arg("build") + .arg("--target-dir") + .arg(&target_dir) + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("lnk-gitd") + .stdout(Stdio::inherit()) + .output() + .expect("cargo build lnk-gitd failed"); + + let child = Command::new(&exec_path) + .arg(lnk_home) + .arg("--push-seeds") + .arg("--fetch-seeds") + .arg("-a") + .arg(addr) + .spawn() + .expect("lnk-gitd failed to start"); + println!("started lnk-gitd"); + child +} + +/// Returns true if runs in the forked child process for git push. +/// returns false if runs in the parent process. +fn run_git_push_in_child_process() -> bool { + let fork = Fork::from_ptmx().unwrap(); + if let Some(mut parent) = fork.is_parent().ok() { + let yes = b"yes\n"; + let buf_reader = BufReader::new(parent); + for line in buf_reader.lines() { + let line = line.unwrap(); + println!("git-push: {}", line); + if line.find("key fingerprint").is_some() { + parent.write_all(yes).unwrap(); + } + } + + false // This is not the child process. + } else { + Command::new("git") + .arg("push") + .arg("linkd") + .status() + .expect("failed to do git push"); + true // This is the child process. + } +} + +fn git_last_commit() -> String { + let output = Command::new("git") + .arg("rev-parse") + .arg("HEAD") + .output() + .expect("failed to run git rev-parse"); + String::from_utf8_lossy(&output.stdout).to_string() +} + +/// Returns UNIX_TIME in millis. +fn timestamp() -> u128 { + let now = SystemTime::now() + .duration_since(SystemTime::UNIX_EPOCH) + .unwrap(); + now.as_millis() +} + +/// Returns the full path of `bins` manifest file. +fn manifest_path() -> String { + let package_dir = env!("CARGO_MANIFEST_DIR"); + format!("{}/Cargo.toml", package_dir.strip_suffix("/tests").unwrap()) +} + +/// Returns the full path of `bins/target`. +fn bins_target_dir() -> String { + let package_dir = env!("CARGO_MANIFEST_DIR"); + format!("{}/target", package_dir.strip_suffix("/tests").unwrap()) +} + +fn clean_up_known_hosts() { + // ssh-keygen -f "/home/pi/.ssh/known_hosts" -R "[127.0.0.1]:9987" + let home_dir = env!("HOME"); + let known_hosts = format!("{}/.ssh/known_hosts", &home_dir); + let output = Command::new("ssh-keygen") + .arg("-f") + .arg(known_hosts) + .arg("-R") + .arg("[127.0.0.1]:9987") + .output() + .expect("failed to do ssh-keygen"); + println!("ssh-keygen: {:?}", &output); +} -- 2.32.0 (Apple Git-132)
Signed-off-by: Han Xu <keepsimple@gmail.com> --- bins/Cargo.lock | 29 +++ bins/Cargo.toml | 1 + bins/tests/Cargo.toml | 13 + bins/tests/integration_test.rs | 456 +++++++++++++++++++++++++++++++++ 4 files changed, 499 insertions(+) create mode 100644 bins/tests/Cargo.toml create mode 100644 bins/tests/integration_test.rs diff --git a/bins/Cargo.lock b/bins/Cargo.lock index ca8d4229..25d0898b 100644 --- a/bins/Cargo.lock @@ -840,6 +840,17 @@ dependencies = [ "termcolor", ] +[[package]] +name = "errno" +version = "0.1.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1e2b2decb0484e15560df3210cf0d78654bb0864b2c138977c07e377a1bae0e2" +dependencies = [ + "kernel32-sys", + "libc", + "winapi 0.2.8", +] + [[package]] name = "event-listener" version = "2.5.3" @@ -2943,6 +2954,16 @@ dependencies = [ "human_format", ] +[[package]] +name = "pty" +version = "0.2.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f50f3d255966981eb4e4c5df3e983e6f7d163221f547406d83b6a460ff5c5ee8" +dependencies = [ + "errno", + "libc", +] + [[package]] name = "quanta" version = "0.4.1" @@ -3568,6 +3589,14 @@ dependencies = [ "winapi-util", ] +[[package]] +name = "tests" +version = "0.1.0" +dependencies = [ + "pty", + "serde_json", +] + [[package]] name = "textwrap" version = "0.15.0" diff --git a/bins/Cargo.toml b/bins/Cargo.toml index 3d1786f4..99b3c69f 100644 --- a/bins/Cargo.toml @@ -5,4 +5,5 @@ members = [ "lnk-gitd", "lnk-identities-dev", "lnk-profile-dev", + "tests", ] diff --git a/bins/tests/Cargo.toml b/bins/tests/Cargo.toml new file mode 100644 index 00000000..3ae82769 --- /dev/null @@ -0,0 +1,13 @@ +[package] +name = "tests" +version = "0.1.0" +authors = ["Han Xu <keepsimple@gmail.com>"] +edition = "2021" + +[dev-dependencies] +pty = "0.2" +serde_json = "1.0" + +[[test]] +name = "integration_test" +path = "integration_test.rs" diff --git a/bins/tests/integration_test.rs b/bins/tests/integration_test.rs new file mode 100644 index 00000000..2d2a4a92 --- /dev/null @@ -0,0 +1,456 @@ +// Copyright © 2022 The Radicle Link Contributors +// +// This file is part of radicle-link, distributed under the GPLv3 with Radicle +// Linking Exception. For full terms see the included LICENSE file. + +//! Integration test to exercise the programs in `bins`. + +use pty::fork::*; +use serde_json::{json, Value}; +use std::{ + env, + fs::File, + io::{BufRead, BufReader, Write}, + process::{Child, Command, Stdio}, + thread, + time::{Duration, SystemTime}, +}; + +/// This test is inspired by https://github.com/alexjg/linkd-playground +/// +/// Tests a typical scenario: there are two peer nodes and one seed node. +/// The main steps are: +/// - Setup a profile for each node in tmp home directories. +/// - Setup SSH keys for each profile. +/// - Create identities. +/// - Create a local repo for peer 1. +/// - Start `linkd` for the seed, and `lnk-gitd` for peer 1. +/// - Push peer 1 repo to its monorepo and to the seed. +/// - Clone the peer 1 repo to peer 2 via seed. +/// +/// This test only works in Linux at this moment. +#[test] +fn two_peers_and_a_seed() { + let peer1_home = "/tmp/link-local-1"; + let peer2_home = "/tmp/link-local-2"; + let seed_home = "/tmp/seed-home"; + let passphrase = b"play\n"; + + { + println!("\n== create lnk homes for two peers and one seed ==\n"); + let cmd = LnkCmd::ProfileCreate; + run_lnk!(&cmd, peer1_home, passphrase); + run_lnk!(&cmd, peer2_home, passphrase); + run_lnk!(&cmd, seed_home, passphrase); + } + + { + println!("\n== add ssh keys for each profile to the ssh-agent ==\n"); + let cmd = LnkCmd::ProfileSshAdd; + run_lnk!(&cmd, peer1_home, passphrase); + run_lnk!(&cmd, peer2_home, passphrase); + run_lnk!(&cmd, seed_home, passphrase); + } + + { + println!("\n== Creating local link 1 identity ==\n"); + let peer1_name = "sockpuppet1".to_string(); + let cmd = LnkCmd::IdPersonCreate { name: peer1_name }; + let output = run_lnk!(&cmd, peer1_home, passphrase); + let v: Value = serde_json::from_str(&output).unwrap(); + let urn = v["urn"].as_str().unwrap().to_string(); + let cmd = LnkCmd::IdLocalSet { urn }; + run_lnk!(&cmd, peer1_home, passphrase); + } + + { + println!("\n== Creating local link 2 identity ==\n"); + let peer2_name = "sockpuppet2".to_string(); + let cmd = LnkCmd::IdPersonCreate { name: peer2_name }; + let output = run_lnk!(&cmd, peer2_home, passphrase); + let v: Value = serde_json::from_str(&output).unwrap(); + let urn = v["urn"].as_str().unwrap().to_string(); + let cmd = LnkCmd::IdLocalSet { urn }; + run_lnk!(&cmd, peer2_home, passphrase); + } + + let peer1_proj_dir = format!("peer1_proj_{}", timestamp()); + let peer1_proj_urn = { + println!("\n== Create a local repository for peer1 ==\n"); + let cmd = LnkCmd::IdProjectCreate { + name: peer1_proj_dir.clone(), + }; + let output = run_lnk!(&cmd, peer1_home, passphrase); + let v: Value = serde_json::from_str(&output).unwrap(); + let proj_urn = v["urn"].as_str().unwrap().to_string(); + println!("our project URN: {}", &proj_urn); + proj_urn + }; + + println!("\n== Add the seed to the local peer seed configs ==\n"); + let seed_endpoint = { + let cmd = LnkCmd::ProfilePeer; + let seed_peer_id = run_lnk!(&cmd, seed_home, passphrase); + format!("{}@127.0.0.1:8799", &seed_peer_id) + }; + + { + // Create seed file for peer1 + let cmd = LnkCmd::ProfileGet; + let peer1_profile = run_lnk!(&cmd, peer1_home, passphrase); + let peer1_seed = format!("{}/{}/seeds", peer1_home, peer1_profile); + let mut peer1_seed_f = File::create(peer1_seed).unwrap(); + peer1_seed_f.write_all(seed_endpoint.as_bytes()).unwrap(); + } + + { + // Create seed file for peer2 + let cmd = LnkCmd::ProfileGet; + let peer2_profile = run_lnk!(&cmd, peer2_home, passphrase); + let peer2_seed = format!("{}/{}/seeds", peer2_home, peer2_profile); + let mut peer2_seed_f = File::create(peer2_seed).unwrap(); + peer2_seed_f.write_all(seed_endpoint.as_bytes()).unwrap(); + } + + println!("\n== Start the seed linkd ==\n"); + let manifest_path = manifest_path(); + let mut linkd = spawn_linkd(seed_home, &manifest_path); + + println!("\n== Start the peer 1 gitd ==\n"); + let cmd = LnkCmd::ProfilePeer; + let gitd_addr = "127.0.0.1:9987"; + let peer1_peer_id = run_lnk!(&cmd, peer1_home, passphrase); + let mut gitd = spawn_lnk_gitd(peer1_home, &manifest_path, gitd_addr); + + println!("\n== Make some changes in the repo: add and commit a test file ==\n"); + env::set_current_dir(&peer1_proj_dir).unwrap(); + let mut test_file = File::create("test").unwrap(); + test_file.write_all(b"test").unwrap(); + Command::new("git") + .arg("add") + .arg("test") + .output() + .expect("failed to do git add"); + let output = Command::new("git") + .arg("commit") + .arg("-m") + .arg("test commit") + .output() + .expect("failed to do git commit"); + println!("git-commit: {:?}", &output); + + println!("\n== Add the linkd remote to the repo ==\n"); + let remote_url = format!("ssh://rad@{}/{}.git", gitd_addr, &peer1_proj_urn); + Command::new("git") + .arg("remote") + .arg("add") + .arg("linkd") + .arg(remote_url) + .output() + .expect("failed to do git remote add"); + + clean_up_known_hosts(); + + if run_git_push_in_child_process() { + // The child process is done with git push. + return; + } + + let peer1_last_commit = git_last_commit(); + + println!("\n== Clone to peer2 ==\n"); + + env::set_current_dir("..").unwrap(); // out of the peer1 proj directory. + let peer2_proj = format!("peer2_proj_{}", timestamp()); + let cmd = LnkCmd::Clone { + urn: peer1_proj_urn, + peer_id: peer1_peer_id, + path: peer2_proj.clone(), + }; + run_lnk!(&cmd, peer2_home, passphrase); + + env::set_current_dir(peer2_proj).unwrap(); + let peer2_last_commit = git_last_commit(); + println!("\n== peer1 proj last commit: {}", &peer1_last_commit); + println!("\n== peer2 proj last commit: {}", &peer2_last_commit); + + println!("\n== Cleanup: kill linkd (seed) and gitd (peer1) ==\n"); + + linkd.kill().ok(); + gitd.kill().ok(); + + assert_eq!(peer1_last_commit, peer2_last_commit); +} + +enum LnkCmd { + ProfileCreate, + ProfileGet, + ProfilePeer, + ProfileSshAdd, + IdPersonCreate { + name: String, + }, + IdLocalSet { + urn: String, + }, + IdProjectCreate { + name: String, + }, + Clone { + urn: String, + peer_id: String, + path: String, + }, +} + +/// Runs a `lnk` command of `$cmd` using `$lnk_home` as the node home. +/// Also support the passphrase input for commands that need it. +#[macro_export] +macro_rules! run_lnk { + ( $cmd:expr, $lnk_home:ident, $passphrase:ident ) => {{ + let fork = Fork::from_ptmx().unwrap(); + if let Some(mut parent) = fork.is_parent().ok() { + match $cmd { + LnkCmd::ProfileCreate | LnkCmd::ProfileSshAdd => { + // Input the passphrase if necessary. + parent.write_all($passphrase).unwrap(); + }, + _ => {}, + } + process_lnk_output($lnk_home, &mut parent, $cmd) + } else { + start_lnk_cmd($lnk_home, $cmd); + return; + } + }}; +} + +fn process_lnk_output(lnk_home: &str, lnk_process: &mut Master, cmd: &LnkCmd) -> String { + let buf_reader = BufReader::new(lnk_process); + let mut output = String::new(); + for line in buf_reader.lines() { + let line = line.unwrap(); + + // Print the output and decode them if necessary. + println!("{}: {}", lnk_home, line); + match cmd { + LnkCmd::IdPersonCreate { name: _ } => { + if line.find("\"urn\":").is_some() { + output = line; // get the line with URN. + } + }, + LnkCmd::IdProjectCreate { name: _ } => { + if line.find("\"urn\":").is_some() { + output = line; // get the line with URN. + } + }, + LnkCmd::ProfileGet => { + output = line; // get the last line for profile id. + }, + LnkCmd::ProfilePeer => { + output = line; // get the last line for peer id. + }, + LnkCmd::Clone { + urn: _, + peer_id: _, + path: _, + } => { + output = line; + }, + _ => {}, + } + } + + output +} + +fn start_lnk_cmd(lnk_home: &str, cmd: &LnkCmd) { + let manifest_path = manifest_path(); + + // cargo run \ + // --manifest-path $LINK_CHECKOUT/bins/Cargo.toml \ + // -p lnk -- "$@" + let mut lnk_cmd = Command::new("cargo"); + lnk_cmd + .env("LNK_HOME", lnk_home) + .arg("run") + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("lnk") + .arg("--"); + let full_cmd = match cmd { + LnkCmd::ProfileCreate => lnk_cmd.arg("profile").arg("create"), + LnkCmd::ProfileGet => lnk_cmd.arg("profile").arg("get"), + LnkCmd::ProfilePeer => lnk_cmd.arg("profile").arg("peer"), + LnkCmd::ProfileSshAdd => lnk_cmd.arg("profile").arg("ssh").arg("add"), + LnkCmd::IdPersonCreate { name } => { + let payload = json!({ "name": name }); + lnk_cmd + .arg("identities") + .arg("person") + .arg("create") + .arg("new") + .arg("--payload") + .arg(payload.to_string()) + }, + LnkCmd::IdLocalSet { urn } => lnk_cmd + .arg("identities") + .arg("local") + .arg("set") + .arg("--urn") + .arg(urn), + LnkCmd::IdProjectCreate { name } => { + let payload = json!({"name": name, "default_branch": "master"}); + let project_path = format!("./{}", name); + lnk_cmd + .arg("identities") + .arg("project") + .arg("create") + .arg("new") + .arg("--path") + .arg(project_path) + .arg("--payload") + .arg(payload.to_string()) + }, + LnkCmd::Clone { urn, peer_id, path } => lnk_cmd + .arg("clone") + .arg("--urn") + .arg(urn) + .arg("--path") + .arg(path) + .arg("--peer") + .arg(peer_id), + }; + full_cmd.status().expect("lnk cmd failed:"); +} + +fn spawn_linkd(lnk_home: &str, manifest_path: &str) -> Child { + let log_name = format!("linkd_{}.log", ×tamp()); + let log_file = File::create(&log_name).unwrap(); + let target_dir = bins_target_dir(); + let exec_path = format!("{}/debug/linkd", &target_dir); + + Command::new("cargo") + .arg("build") + .arg("--target-dir") + .arg(&target_dir) + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("linkd") + .output() + .expect("cargo build linkd failed"); + + let child = Command::new(&exec_path) + .env("RUST_BACKTRACE", "1") + .arg("--lnk-home") + .arg(lnk_home) + .arg("--track") + .arg("everything") + .arg("--protocol-listen") + .arg("127.0.0.1:8799") + .stdout(Stdio::from(log_file)) + .spawn() + .expect("linkd failed to start"); + println!("linkd stdout redirected to {}", &log_name); + thread::sleep(Duration::from_secs(1)); + child +} + +fn spawn_lnk_gitd(lnk_home: &str, manifest_path: &str, addr: &str) -> Child { + let target_dir = bins_target_dir(); + let exec_path = format!("{}/debug/lnk-gitd", &target_dir); + + Command::new("cargo") + .arg("build") + .arg("--target-dir") + .arg(&target_dir) + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("lnk-gitd") + .stdout(Stdio::inherit()) + .output() + .expect("cargo build lnk-gitd failed"); + + let child = Command::new(&exec_path) + .arg(lnk_home) + .arg("--push-seeds") + .arg("--fetch-seeds") + .arg("-a") + .arg(addr) + .spawn() + .expect("lnk-gitd failed to start"); + println!("started lnk-gitd"); + child +} + +/// Returns true if runs in the forked child process for git push. +/// returns false if runs in the parent process. +fn run_git_push_in_child_process() -> bool { + let fork = Fork::from_ptmx().unwrap(); + if let Some(mut parent) = fork.is_parent().ok() { + let yes = b"yes\n"; + let buf_reader = BufReader::new(parent); + for line in buf_reader.lines() { + let line = line.unwrap(); + println!("git-push: {}", line); + if line.find("key fingerprint").is_some() { + parent.write_all(yes).unwrap(); + } + } + + false // This is not the child process. + } else { + Command::new("git") + .arg("push") + .arg("linkd") + .status() + .expect("failed to do git push"); + true // This is the child process. + } +} + +fn git_last_commit() -> String { + let output = Command::new("git") + .arg("rev-parse") + .arg("HEAD") + .output() + .expect("failed to run git rev-parse"); + String::from_utf8_lossy(&output.stdout).to_string() +} + +/// Returns UNIX_TIME in millis. +fn timestamp() -> u128 { + let now = SystemTime::now() + .duration_since(SystemTime::UNIX_EPOCH) + .unwrap(); + now.as_millis() +} + +/// Returns the full path of `bins` manifest file. +fn manifest_path() -> String { + let package_dir = env!("CARGO_MANIFEST_DIR"); + format!("{}/Cargo.toml", package_dir.strip_suffix("/tests").unwrap()) +} + +/// Returns the full path of `bins/target`. +fn bins_target_dir() -> String { + let package_dir = env!("CARGO_MANIFEST_DIR"); + format!("{}/target", package_dir.strip_suffix("/tests").unwrap()) +} + +fn clean_up_known_hosts() { + // ssh-keygen -f "/home/pi/.ssh/known_hosts" -R "[127.0.0.1]:9987" + let home_dir = env!("HOME"); + let known_hosts = format!("{}/.ssh/known_hosts", &home_dir); + let output = Command::new("ssh-keygen") + .arg("-f") + .arg(known_hosts) + .arg("-R") + .arg("[127.0.0.1]:9987") + .output() + .expect("failed to do ssh-keygen"); + println!("ssh-keygen: {:?}", &output); +} -- 2.32.1 (Apple Git-133)
--- cli/gitd-lib/src/hooks.rs | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/cli/gitd-lib/src/hooks.rs b/cli/gitd-lib/src/hooks.rs index 5e928ffd..cf37d71f 100644 --- a/cli/gitd-lib/src/hooks.rs +++ b/cli/gitd-lib/src/hooks.rs @@ -77,6 +77,16 @@ where E: std::error::Error + Send + 'static, P: ProgressReporter<Error = E>, { + // update the signed refs before a possible request_pull, + // so that the peer can receive the latest refs. + let at = update_signed_refs( + reporter, + self.spawner.clone(), + self.pool.clone(), + urn.clone(), + ) + .await?; + if self.post_receive.request_pull { tracing::info!("executing request-pull"); request_pull(reporter, &self.client, &self.seeds, urn.clone()).await?; @@ -87,14 +97,7 @@ where ) .await?; } - let at = match update_signed_refs( - reporter, - self.spawner.clone(), - self.pool.clone(), - urn.clone(), - ) - .await? - { + let at = match at { Some(at) => at, None => return Ok(()), }; -- 2.32.0 (Apple Git-132)
--- cli/gitd-lib/src/hooks.rs | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/cli/gitd-lib/src/hooks.rs b/cli/gitd-lib/src/hooks.rs index 5e928ffd..cf37d71f 100644 --- a/cli/gitd-lib/src/hooks.rs +++ b/cli/gitd-lib/src/hooks.rs @@ -77,6 +77,16 @@ where E: std::error::Error + Send + 'static, P: ProgressReporter<Error = E>, { + // update the signed refs before a possible request_pull, + // so that the peer can receive the latest refs. + let at = update_signed_refs( + reporter, + self.spawner.clone(), + self.pool.clone(), + urn.clone(), + ) + .await?; + if self.post_receive.request_pull { tracing::info!("executing request-pull"); request_pull(reporter, &self.client, &self.seeds, urn.clone()).await?; @@ -87,14 +97,7 @@ where ) .await?; } - let at = match update_signed_refs( - reporter, - self.spawner.clone(), - self.pool.clone(), - urn.clone(), - ) - .await? - { + let at = match at { Some(at) => at, None => return Ok(()), }; -- 2.32.0 (Apple Git-132)
--- link-git/src/protocol/ls.rs | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/link-git/src/protocol/ls.rs b/link-git/src/protocol/ls.rs index b3516454..de95b4a3 100644 --- a/link-git/src/protocol/ls.rs +++ b/link-git/src/protocol/ls.rs @@ -22,9 +22,12 @@ use super::{remote_git_version, transport}; // Work around `git-upload-pack` not handling namespaces properly // // cf. https://lore.kernel.org/git/pMV5dJabxOBTD8kJBaPuWK0aS6OJhRQ7YFGwfhPCeSJEbPDrIFBza36nXBCgUCeUJWGmpjPI1rlOGvZJEh71Ruz4SqljndUwOCoBUDRHRDU=@eagain.st/ +// +// Based on testing with git 2.25.1 in Ubuntu 20.04, this workaround is +// not needed. Hence the checked version is lowered to 2.25.0. fn must_namespace(caps: &client::Capabilities) -> bool { static MIN_GIT_VERSION_NAMESPACES: Lazy<Version> = - Lazy::new(|| Version::new("2.31.0").unwrap()); + Lazy::new(|| Version::new("2.25.0").unwrap()); remote_git_version(caps) .map(|version| version < *MIN_GIT_VERSION_NAMESPACES) -- 2.32.0 (Apple Git-132)
Signed-off-by: Han Xu <keepsimple@gmail.com> --- link-git/src/protocol/ls.rs | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/link-git/src/protocol/ls.rs b/link-git/src/protocol/ls.rs index b3516454..de95b4a3 100644 --- a/link-git/src/protocol/ls.rs +++ b/link-git/src/protocol/ls.rs @@ -22,9 +22,12 @@ use super::{remote_git_version, transport}; // Work around `git-upload-pack` not handling namespaces properly // // cf. https://lore.kernel.org/git/pMV5dJabxOBTD8kJBaPuWK0aS6OJhRQ7YFGwfhPCeSJEbPDrIFBza36nXBCgUCeUJWGmpjPI1rlOGvZJEh71Ruz4SqljndUwOCoBUDRHRDU=@eagain.st/ +// +// Based on testing with git 2.25.1 in Ubuntu 20.04, this workaround is +// not needed. Hence the checked version is lowered to 2.25.0. fn must_namespace(caps: &client::Capabilities) -> bool { static MIN_GIT_VERSION_NAMESPACES: Lazy<Version> = - Lazy::new(|| Version::new("2.31.0").unwrap()); + Lazy::new(|| Version::new("2.25.0").unwrap()); remote_git_version(caps) .map(|version| version < *MIN_GIT_VERSION_NAMESPACES) -- 2.32.1 (Apple Git-133)
--- bins/tests/integration_test.rs | 353 +++++++++++++++------------------ 1 file changed, 158 insertions(+), 195 deletions(-) diff --git a/bins/tests/integration_test.rs b/bins/tests/integration_test.rs index ea6815ab..43de968a 100644 --- a/bins/tests/integration_test.rs @@ -35,92 +35,55 @@ fn two_peers_and_a_seed() { let passphrase = b"play\n"; println!("\n== create lnk homes for two peers and one seed ==\n"); - let (is_parent, _) = run_lnk(LnkCmd::ProfileCreate, peer1_home, passphrase); - if !is_parent { - return; - } - let (is_parent, _) = run_lnk(LnkCmd::ProfileCreate, peer2_home, passphrase); - if !is_parent { - return; - } - let (is_parent, _) = run_lnk(LnkCmd::ProfileCreate, seed_home, passphrase); - if !is_parent { - return; - } + let cmd = LnkCmd::ProfileCreate; + run_lnk!(&cmd, peer1_home, passphrase); + run_lnk!(&cmd, peer2_home, passphrase); + run_lnk!(&cmd, seed_home, passphrase); println!("\n== add ssh keys for each profile to the ssh-agent ==\n"); - let (is_parent, _) = run_lnk(LnkCmd::ProfileSshAdd, peer1_home, passphrase); - if !is_parent { - return; - } - let (is_parent, _) = run_lnk(LnkCmd::ProfileSshAdd, peer2_home, passphrase); - if !is_parent { - return; - } - let (is_parent, _) = run_lnk(LnkCmd::ProfileSshAdd, seed_home, passphrase); - if !is_parent { - return; - } + let cmd = LnkCmd::ProfileSshAdd; + run_lnk!(&cmd, peer1_home, passphrase); + run_lnk!(&cmd, peer2_home, passphrase); + run_lnk!(&cmd, seed_home, passphrase); println!("\n== Creating local link 1 identity ==\n"); let peer1_name = "sockpuppet1".to_string(); - let (is_parent, output) = run_lnk(LnkCmd::IdPersonCreate(peer1_name), peer1_home, passphrase); - if !is_parent { - return; - } + let cmd = LnkCmd::IdPersonCreate(peer1_name); + let output = run_lnk!(&cmd, peer1_home, passphrase); let v: Value = serde_json::from_str(&output).unwrap(); let urn1 = v["urn"].as_str().unwrap().to_string(); - let (is_parent, _) = run_lnk(LnkCmd::IdLocalSet(urn1), peer1_home, passphrase); - if !is_parent { - return; - } + let cmd = LnkCmd::IdLocalSet(urn1); + run_lnk!(&cmd, peer1_home, passphrase); println!("\n== Creating local link 2 identity ==\n"); let peer2_name = "sockpuppet2".to_string(); - let (is_parent, output) = run_lnk(LnkCmd::IdPersonCreate(peer2_name), peer2_home, passphrase); - if !is_parent { - return; - } + let cmd = LnkCmd::IdPersonCreate(peer2_name); + let output = run_lnk!(&cmd, peer2_home, passphrase); let v: Value = serde_json::from_str(&output).unwrap(); let urn2 = v["urn"].as_str().unwrap().to_string(); - let (is_parent, _) = run_lnk(LnkCmd::IdLocalSet(urn2), peer2_home, passphrase); - if !is_parent { - return; - } + let cmd = LnkCmd::IdLocalSet(urn2); + run_lnk!(&cmd, peer2_home, passphrase); println!("\n== Create a local repository ==\n"); let peer1_proj = format!("peer1_proj_{}", timestamp()); - let (is_parent, output) = run_lnk( - LnkCmd::IdProjectCreate(peer1_proj.clone()), - peer1_home, - passphrase, - ); - if !is_parent { - return; - } + let cmd = LnkCmd::IdProjectCreate(peer1_proj.clone()); + let output = run_lnk!(&cmd, peer1_home, passphrase); let v: Value = serde_json::from_str(&output).unwrap(); let proj_urn = v["urn"].as_str().unwrap().to_string(); println!("our project URN: {}", &proj_urn); println!("\n== Add the seed to the local peer seed configs ==\n"); - let (is_parent, seed_peer_id) = run_lnk(LnkCmd::ProfilePeer, seed_home, passphrase); - if !is_parent { - return; - } + let cmd = LnkCmd::ProfilePeer; + let seed_peer_id = run_lnk!(&cmd, seed_home, passphrase); let seed_endpoint = format!("{}@127.0.0.1:8799", &seed_peer_id); - let (is_parent, peer1_profile) = run_lnk(LnkCmd::ProfileGet, peer1_home, passphrase); - if !is_parent { - return; - } + let cmd = LnkCmd::ProfileGet; + let peer1_profile = run_lnk!(&cmd, peer1_home, passphrase); let peer1_seed = format!("{}/{}/seeds", peer1_home, peer1_profile); let mut peer1_f = File::create(peer1_seed).unwrap(); peer1_f.write_all(seed_endpoint.as_bytes()).unwrap(); - let (is_parent, peer2_profile) = run_lnk(LnkCmd::ProfileGet, peer2_home, passphrase); - if !is_parent { - return; - } + let peer2_profile = run_lnk!(&cmd, peer2_home, passphrase); let peer2_seed = format!("{}/{}/seeds", peer2_home, peer2_profile); let mut peer2_f = File::create(peer2_seed).unwrap(); peer2_f.write_all(seed_endpoint.as_bytes()).unwrap(); @@ -130,10 +93,8 @@ fn two_peers_and_a_seed() { let mut linkd = spawn_linkd(seed_home, &manifest_path); println!("\n== Start the peer 1 gitd ==\n"); - let (is_parent, peer1_peer_id) = run_lnk(LnkCmd::ProfilePeer, peer1_home, passphrase); - if !is_parent { - return; - } + let cmd = LnkCmd::ProfilePeer; + let peer1_peer_id = run_lnk!(&cmd, peer1_home, passphrase); spawn_lnk_gitd(peer1_home, &manifest_path, &peer1_peer_id); println!("\n== Make some changes in the repo ==\n"); @@ -171,17 +132,17 @@ fn two_peers_and_a_seed() { } let peer1_last_commit = git_last_commit(); - println!("\n== peer1 project last commit: {} ==\n", &peer1_last_commit); + println!( + "\n== peer1 project last commit: {} ==\n", + &peer1_last_commit + ); println!("\n== Clone to peer2 ==\n"); env::set_current_dir("..").unwrap(); // out of the peer1 proj directory. let peer2_proj = format!("peer2_proj_{}", timestamp()); - let (is_parent, _) = run_lnk( - LnkCmd::Clone(proj_urn, peer1_peer_id, peer2_proj.clone()), - peer2_home, - passphrase, - ); + let cmd = LnkCmd::Clone(proj_urn, peer1_peer_id, peer2_proj.clone()); + run_lnk!(&cmd, peer2_home, passphrase); if !is_parent { return; } @@ -206,122 +167,125 @@ enum LnkCmd { IdPersonCreate(String), // the associated string is "the person's name". IdLocalSet(String), // the associated string is "urn". IdProjectCreate(String), // the associated string is "the project name". - Clone(String, String, String), // the associated string is "urn", "peer_id", "path" + Clone(String, String, String), // the associated string is "urn", "peer_id", "path" } -/// Runs a `cmd` for `lnk_home`. Rebuilds `lnk` if necessary. -/// Return.0: true if this is the parent (i.e. test) process, -/// false if this is the child (i.e. lnk) process. -/// Return.1: an output that depends on the `cmd`. -fn run_lnk(cmd: LnkCmd, lnk_home: &str, passphrase: &[u8]) -> (bool, String) { - let fork = Fork::from_ptmx().unwrap(); - if let Some(mut parent) = fork.is_parent().ok() { - // Input the passphrase if necessary. - match cmd { - LnkCmd::ProfileCreate | LnkCmd::ProfileSshAdd => { - parent.write_all(passphrase).unwrap(); - println!("{}: wrote passphase", lnk_home); - }, - _ => {}, - } - - // Print the output and decode them if necessary. - let buf_reader = BufReader::new(parent); - let mut output = String::new(); - for line in buf_reader.lines() { - let line = line.unwrap(); - println!("{}: {}", lnk_home, line); - - match cmd { - LnkCmd::IdPersonCreate(ref _name) => { - if line.find("\"urn\":").is_some() { - output = line; // get the line with URN. - } - }, - LnkCmd::IdProjectCreate(ref _name) => { - if line.find("\"urn\":").is_some() { - output = line; // get the line with URN. - } - }, - LnkCmd::ProfileGet => { - output = line; // get the last line for profile id. - }, - LnkCmd::ProfilePeer => { - output = line; // get the last line for peer id. - }, - LnkCmd::Clone(ref _urn, ref _peer, ref _path) => { - output = line; +/// Runs a `lnk` command of `$cmd` using `$lnk_home` as the node home. +/// Also support the passphrase input for commands that need it. +#[macro_export] +macro_rules! run_lnk { + ( $cmd:expr, $lnk_home:ident, $passphrase:ident ) => {{ + let fork = Fork::from_ptmx().unwrap(); + if let Some(mut parent) = fork.is_parent().ok() { + match $cmd { + LnkCmd::ProfileCreate | LnkCmd::ProfileSshAdd => { + // Input the passphrase if necessary. + parent.write_all($passphrase).unwrap(); }, _ => {}, } + process_lnk_output($lnk_home, &mut parent, $cmd) + } else { + start_lnk_cmd($lnk_home, $cmd); + return; } + }}; +} - (true, output) - } else { - // Child process is to run `lnk`. - let manifest_path = manifest_path(); - - // cargo run \ - // --manifest-path $LINK_CHECKOUT/bins/Cargo.toml \ - // -p lnk -- "$@" - let mut lnk_cmd = Command::new("cargo"); - lnk_cmd - .env("LNK_HOME", lnk_home) - .arg("run") - .arg("--manifest-path") - .arg(manifest_path) - .arg("-p") - .arg("lnk") - .arg("--"); - let full_cmd = match cmd { - LnkCmd::ProfileCreate => lnk_cmd.arg("profile").arg("create"), - LnkCmd::ProfileGet => lnk_cmd.arg("profile").arg("get"), - LnkCmd::ProfilePeer => lnk_cmd.arg("profile").arg("peer"), - LnkCmd::ProfileSshAdd => lnk_cmd.arg("profile").arg("ssh").arg("add"), - LnkCmd::IdPersonCreate(name) => { - let payload = json!({ "name": name }); - lnk_cmd - .arg("identities") - .arg("person") - .arg("create") - .arg("new") - .arg("--payload") - .arg(payload.to_string()) +fn process_lnk_output(lnk_home: &str, lnk_process: &mut Master, cmd: &LnkCmd) -> String { + let buf_reader = BufReader::new(lnk_process); + let mut output = String::new(); + for line in buf_reader.lines() { + let line = line.unwrap(); + + // Print the output and decode them if necessary. + println!("{}: {}", lnk_home, line); + match cmd { + LnkCmd::IdPersonCreate(ref _name) => { + if line.find("\"urn\":").is_some() { + output = line; // get the line with URN. + } }, - LnkCmd::IdLocalSet(urn) => lnk_cmd - .arg("identities") - .arg("local") - .arg("set") - .arg("--urn") - .arg(urn), - LnkCmd::IdProjectCreate(name) => { - let payload = json!({"name": name, "default_branch": "master"}); - let project_path = format!("./{}", name); - lnk_cmd - .arg("identities") - .arg("project") - .arg("create") - .arg("new") - .arg("--path") - .arg(project_path) - .arg("--payload") - .arg(payload.to_string()) + LnkCmd::IdProjectCreate(ref _name) => { + if line.find("\"urn\":").is_some() { + output = line; // get the line with URN. + } }, - LnkCmd::Clone(urn, peer_id, peer2_proj) => { - lnk_cmd - .arg("clone") - .arg("--urn") - .arg(urn) - .arg("--path") - .arg(peer2_proj) - .arg("--peer") - .arg(peer_id) + LnkCmd::ProfileGet => { + output = line; // get the last line for profile id. }, - }; - full_cmd.status().expect("lnk cmd failed:"); - - (false, String::new()) + LnkCmd::ProfilePeer => { + output = line; // get the last line for peer id. + }, + LnkCmd::Clone(ref _urn, ref _peer, ref _path) => { + output = line; + }, + _ => {}, + } } + + output +} + +fn start_lnk_cmd(lnk_home: &str, cmd: &LnkCmd) { + let manifest_path = manifest_path(); + + // cargo run \ + // --manifest-path $LINK_CHECKOUT/bins/Cargo.toml \ + // -p lnk -- "$@" + let mut lnk_cmd = Command::new("cargo"); + lnk_cmd + .env("LNK_HOME", lnk_home) + .arg("run") + .arg("--manifest-path") + .arg(manifest_path) + .arg("-p") + .arg("lnk") + .arg("--"); + let full_cmd = match cmd { + LnkCmd::ProfileCreate => lnk_cmd.arg("profile").arg("create"), + LnkCmd::ProfileGet => lnk_cmd.arg("profile").arg("get"), + LnkCmd::ProfilePeer => lnk_cmd.arg("profile").arg("peer"), + LnkCmd::ProfileSshAdd => lnk_cmd.arg("profile").arg("ssh").arg("add"), + LnkCmd::IdPersonCreate(name) => { + let payload = json!({ "name": name }); + lnk_cmd + .arg("identities") + .arg("person") + .arg("create") + .arg("new") + .arg("--payload") + .arg(payload.to_string()) + }, + LnkCmd::IdLocalSet(urn) => lnk_cmd + .arg("identities") + .arg("local") + .arg("set") + .arg("--urn") + .arg(urn), + LnkCmd::IdProjectCreate(name) => { + let payload = json!({"name": name, "default_branch": "master"}); + let project_path = format!("./{}", name); + lnk_cmd + .arg("identities") + .arg("project") + .arg("create") + .arg("new") + .arg("--path") + .arg(project_path) + .arg("--payload") + .arg(payload.to_string()) + }, + LnkCmd::Clone(urn, peer_id, peer2_proj) => lnk_cmd + .arg("clone") + .arg("--urn") + .arg(urn) + .arg("--path") + .arg(peer2_proj) + .arg("--peer") + .arg(peer_id), + }; + full_cmd.status().expect("lnk cmd failed:"); } fn spawn_linkd(lnk_home: &str, manifest_path: &str) -> Child { @@ -376,26 +340,25 @@ fn spawn_lnk_gitd(lnk_home: &str, manifest_path: &str, peer_id: &str) { .output() .expect("cargo build lnk-gitd failed"); - Command::new("systemd-socket-activate") - .arg("-l") - .arg(port) - .arg("--fdname=ssh") - .arg("-E") - .arg("SSH_AUTH_SOCK") - .arg("-E") - .arg("RUST_BACKTRACE") - .arg(&exec_path) - .arg(lnk_home) - .arg("--linkd-rpc-socket") - .arg(rpc_socket) - .arg("--push-seeds") - .arg("--fetch-seeds") - .arg("--linger-timeout") - .arg("10000") - .spawn() - .expect("lnk-gitd failed to start"); - println!("started lnk-gitd"); - + Command::new("systemd-socket-activate") + .arg("-l") + .arg(port) + .arg("--fdname=ssh") + .arg("-E") + .arg("SSH_AUTH_SOCK") + .arg("-E") + .arg("RUST_BACKTRACE") + .arg(&exec_path) + .arg(lnk_home) + .arg("--linkd-rpc-socket") + .arg(rpc_socket) + .arg("--push-seeds") + .arg("--fetch-seeds") + .arg("--linger-timeout") + .arg("10000") + .spawn() + .expect("lnk-gitd failed to start"); + println!("started lnk-gitd"); } /// Returns true if this is the parent process, -- 2.32.0 (Apple Git-132)
--- bins/tests/integration_test.rs | 34 ++++++++++------------------------ 1 file changed, 10 insertions(+), 24 deletions(-) diff --git a/bins/tests/integration_test.rs b/bins/tests/integration_test.rs index 43de968a..d245748d 100644 --- a/bins/tests/integration_test.rs @@ -94,8 +94,9 @@ fn two_peers_and_a_seed() { println!("\n== Start the peer 1 gitd ==\n"); let cmd = LnkCmd::ProfilePeer; + let gitd_addr = "127.0.0.1:9987"; let peer1_peer_id = run_lnk!(&cmd, peer1_home, passphrase); - spawn_lnk_gitd(peer1_home, &manifest_path, &peer1_peer_id); + let mut gitd = spawn_lnk_gitd(peer1_home, &manifest_path, gitd_addr); println!("\n== Make some changes in the repo ==\n"); env::set_current_dir(&peer1_proj).unwrap(); @@ -115,7 +116,7 @@ fn two_peers_and_a_seed() { println!("git-commit: {:?}", &output); println!("\n== Add the linkd remote to the repo ==\n"); - let remote_url = format!("ssh://rad@127.0.0.1:9987/{}.git", &proj_urn); + let remote_url = format!("ssh://rad@{}/{}.git", gitd_addr, &proj_urn); Command::new("git") .arg("remote") .arg("add") @@ -132,10 +133,6 @@ fn two_peers_and_a_seed() { } let peer1_last_commit = git_last_commit(); - println!( - "\n== peer1 project last commit: {} ==\n", - &peer1_last_commit - ); println!("\n== Clone to peer2 ==\n"); @@ -152,9 +149,10 @@ fn two_peers_and_a_seed() { println!("\n== peer1 proj last commit: {}", &peer1_last_commit); println!("\n== peer2 proj last commit: {}", &peer2_last_commit); - println!("\n== Cleanup: kill linkd (seed) ==\n"); + println!("\n== Cleanup: kill linkd (seed) and gitd (peer1) ==\n"); linkd.kill().ok(); + gitd.kill().ok(); assert_eq!(peer1_last_commit, peer2_last_commit); } @@ -321,10 +319,7 @@ fn spawn_linkd(lnk_home: &str, manifest_path: &str) -> Child { child } -fn spawn_lnk_gitd(lnk_home: &str, manifest_path: &str, peer_id: &str) { - let port = "9987"; - let xdg_runtime_dir = env!("XDG_RUNTIME_DIR"); - let rpc_socket = format!("{}/link-peer-{}-rpc.socket", xdg_runtime_dir, peer_id); +fn spawn_lnk_gitd(lnk_home: &str, manifest_path: &str, addr: &str) -> Child { let target_dir = bins_target_dir(); let exec_path = format!("{}/debug/lnk-gitd", &target_dir); @@ -340,25 +335,16 @@ fn spawn_lnk_gitd(lnk_home: &str, manifest_path: &str, peer_id: &str) { .output() .expect("cargo build lnk-gitd failed"); - Command::new("systemd-socket-activate") - .arg("-l") - .arg(port) - .arg("--fdname=ssh") - .arg("-E") - .arg("SSH_AUTH_SOCK") - .arg("-E") - .arg("RUST_BACKTRACE") - .arg(&exec_path) + let child = Command::new(&exec_path) .arg(lnk_home) - .arg("--linkd-rpc-socket") - .arg(rpc_socket) .arg("--push-seeds") .arg("--fetch-seeds") - .arg("--linger-timeout") - .arg("10000") + .arg("-a") + .arg(addr) .spawn() .expect("lnk-gitd failed to start"); println!("started lnk-gitd"); + child } /// Returns true if this is the parent process, -- 2.32.0 (Apple Git-132)