~rjarry/dlrepo

dlrepo: cli: remove unused local_dir() function v1 NEEDS REVISION

Julien Floret: 7
 cli: remove unused local_dir() function
 cli: remove unused local_dir() function
 fmt: remove internal flag
 job: add internal flag
 job: display upload time
 tag: clean published jobs before re-publishing
 tag: robustify TagView.put()

 33 files changed, 205 insertions(+), 152 deletions(-)
#754971 .build.yml success
Le mar. 10 mai 2022 à 18:19, Olivier Matz <zer0@droids-corp.org> a écrit :
Next
Export patchset (mbox)
How do I use this?

Copy & paste the following snippet into your terminal to import this patchset into git:

curl -s https://lists.sr.ht/~rjarry/dlrepo/patches/32147/mbox | git am -3
Learn more about email & git

[PATCH dlrepo] cli: remove unused local_dir() function Export this patch

Signed-off-by: Julien Floret <julien.floret@6wind.com>
---
 dlrepo-cli | 7 -------
 1 file changed, 7 deletions(-)

diff --git a/dlrepo-cli b/dlrepo-cli
index 298387df9631..476d3350ad29 100755
--- a/dlrepo-cli
+++ b/dlrepo-cli
@@ -131,13 +131,6 @@ def local_path(value):
    return value


# --------------------------------------------------------------------------------------
def local_dir(value):
    if not os.path.isdir(value):
        raise argparse.ArgumentTypeError("%r: No such directory" % (value,))
    return value


# --------------------------------------------------------------------------------------
def job_param(value):
    match = re.match(r"^(\w+)=(.*)$", value)
-- 
2.30.2
dlrepo/patches/.build.yml: SUCCESS in 1m31s

[cli: remove unused local_dir() function][0] from [Julien Floret][1]

[0]: https://lists.sr.ht/~rjarry/dlrepo/patches/32147
[1]: mailto:julien.floret@6wind.com

✓ #754971 SUCCESS dlrepo/patches/.build.yml https://builds.sr.ht/~rjarry/job/754971
The "internal_job" job is flagged "internal". Test that it is not
published to the remote server.

Signed-off-by: Julien Floret <julien.floret@6wind.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>
---
 .../branches/branch/tag/internal_job/.internal               | 0
 .../branches/branch/tag/internal_job/fmt1/.digests           | 1 +
 .../branches/branch/tag/internal_job/fmt1/a                  | 1 +
 .../branches/branch/tag/internal_job/fmt1/b                  | 1 +
 tests/test_publish.py                                        | 5 +++++
 5 files changed, 8 insertions(+)
 create mode 100644 tests/publish_reference/branches/branch/tag/internal_job/.internal
 create mode 100644 tests/publish_reference/branches/branch/tag/internal_job/fmt1/.digests
 create mode 100644 tests/publish_reference/branches/branch/tag/internal_job/fmt1/a
 create mode 100644 tests/publish_reference/branches/branch/tag/internal_job/fmt1/b

diff --git a/tests/publish_reference/branches/branch/tag/internal_job/.internal b/tests/publish_reference/branches/branch/tag/internal_job/.internal
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tests/publish_reference/branches/branch/tag/internal_job/fmt1/.digests b/tests/publish_reference/branches/branch/tag/internal_job/fmt1/.digests
new file mode 100644
index 000000000000..7ae13f34b9e3
--- /dev/null
+++ b/tests/publish_reference/branches/branch/tag/internal_job/fmt1/.digests
@@ -0,0 +1 @@
+{"b": "sha256:0263829989b6fd954f72baaf2fc64bc2e2f01d692d4de72986ea808f6e99813f", "a": "sha256:87428fc522803d31065e7bce3cf03fe475096631e5e07bbd7a0fde60c4cf25c7"}
\ No newline at end of file
diff --git a/tests/publish_reference/branches/branch/tag/internal_job/fmt1/a b/tests/publish_reference/branches/branch/tag/internal_job/fmt1/a
new file mode 100644
index 000000000000..78981922613b
--- /dev/null
+++ b/tests/publish_reference/branches/branch/tag/internal_job/fmt1/a
@@ -0,0 +1 @@
+a
diff --git a/tests/publish_reference/branches/branch/tag/internal_job/fmt1/b b/tests/publish_reference/branches/branch/tag/internal_job/fmt1/b
new file mode 100644
index 000000000000..61780798228d
--- /dev/null
+++ b/tests/publish_reference/branches/branch/tag/internal_job/fmt1/b
@@ -0,0 +1 @@
+b
diff --git a/tests/test_publish.py b/tests/test_publish.py
index cceb07c97695..7c9f2f572749 100644
--- a/tests/test_publish.py
+++ b/tests/test_publish.py
@@ -83,3 +83,8 @@ async def test_publish(dlrepo_servers):  # pylint: disable=redefined-outer-name
                     assert resp.status == 200
                     data_pub = await resp.text()
                     assert data_pub == data
+            sha_url = "/branches/branch/tag/internal_job/fmt1.sha256"
+            resp = await sess.get(sha_url)
+            assert resp.status == 200
+            resp = await pub_sess.get(sha_url)
+            assert resp.status == 404
-- 
2.30.2

[PATCH dlrepo v2 1/7] cli: remove unused local_dir() function Export this patch

This function is present since the import of the file but has never been
used.

Fixes: 49ff6a714f43 ("Add client script")
Signed-off-by: Julien Floret <julien.floret@6wind.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>
---
Changes v1 -> v2:
 - Added "Fixes" lign.

 dlrepo-cli | 7 -------
 1 file changed, 7 deletions(-)

diff --git a/dlrepo-cli b/dlrepo-cli
index 298387df9631..476d3350ad29 100755
--- a/dlrepo-cli
+++ b/dlrepo-cli
@@ -131,13 +131,6 @@ def local_path(value):
    return value


# --------------------------------------------------------------------------------------
def local_dir(value):
    if not os.path.isdir(value):
        raise argparse.ArgumentTypeError("%r: No such directory" % (value,))
    return value


# --------------------------------------------------------------------------------------
def job_param(value):
    match = re.match(r"^(\w+)=(.*)$", value)
-- 
2.30.2

[PATCH dlrepo v2 2/7] fmt: remove internal flag Export this patch

The "internal" flag on artifacts formats can be quite misleading,
because it is only informative and does not prevent the format to be
uploaded to a remote publish server. ACLs should be more than enough to
hide some formats if need be.

Note: after upgrading your server, you must remove the ".internal"
files manually from your server artifact database:

  # rm -f $DLREPO_ROOT_DIR/branches/*/*/*/*/.internal

Signed-off-by: Julien Floret <julien.floret@6wind.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>
---
This one is a new patch. In the next patch we will introduce the
"internal" flag for jobs, which would have been confusing with the
format's "internal" flag, because it does not have the same meaning.
As discussed in the ML, we decided to remove the format's "internal"
flag, which is only informative and does not bring much added value.

 dlrepo-cli                            | 22 ----------------------
 dlrepo/fs/fmt.py                      | 17 +----------------
 dlrepo/fs/tag.py                      |  5 -----
 dlrepo/templates/job.html             |  5 +----
 dlrepo/templates/product_version.html |  5 +----
 dlrepo/views/fmt.py                   | 20 --------------------
 dlrepo/views/job.py                   |  1 -
 dlrepo/views/product.py               |  1 -
 dlrepo/views/util.py                  |  1 -
 docs/dlrepo-api.7.scdoc               | 25 -------------------------
 docs/dlrepo-cli.1.scdoc               | 17 -----------------
 docs/dlrepo-layout.7.scdoc            |  3 ---
 scss/sections.scss                    |  6 ------
 13 files changed, 3 insertions(+), 125 deletions(-)

diff --git a/dlrepo-cli b/dlrepo-cli
index 476d3350ad29..351140ecabb0 100755
--- a/dlrepo-cli
+++ b/dlrepo-cli
@@ -386,28 +386,6 @@ def release(args):
    client.put(url, {"tag": {"released": not args.unset}})


# --------------------------------------------------------------------------------------
@sub_command(
    Arg("branch", metavar="BRANCH", help="the branch name"),
    Arg("tag", metavar="TAG", help="the tag name"),
    Arg("job", metavar="JOB", help="the job name"),
    Arg("format", metavar="FORMAT", help="the artifact format"),
    Arg(
        "-u",
        "--unset",
        action="store_true",
        help="unset the 'internal' status from the format instead of setting it",
    ),
)
def internal(args):
    """
    Set or unset the 'internal' status on an artifact format.
    """
    client = HttpClient(args.url)
    url = os.path.join("branches", args.branch, args.tag, args.job, args.format) + "/"
    client.put(url, {"artifact_format": {"internal": not args.unset}})


# --------------------------------------------------------------------------------------
@sub_command(
    Arg("branch", metavar="BRANCH", help="the branch name"),
diff --git a/dlrepo/fs/fmt.py b/dlrepo/fs/fmt.py
index a58602198d0b..29c372e0627e 100644
--- a/dlrepo/fs/fmt.py
+++ b/dlrepo/fs/fmt.py
@@ -44,14 +44,12 @@ class ArtifactFormat(SubDir):

    @cachedmethod(lambda self: self._is_reserved_cache)
    def _is_reserved_file(self, path, *, resolve=False):
        internal = self._internal_path()
        digests = self._digest_path()
        dirty = self._dirty_path()
        if resolve:
            internal = internal.resolve()
            digests = digests.resolve()
            dirty = dirty.resolve()
        return path in (internal, digests, dirty)
        return path in (digests, dirty)

    def list_dir(self, relpath: str) -> Tuple[List[str], List[str]]:
        path = self.get_filepath(relpath)
@@ -126,19 +124,6 @@ class ArtifactFormat(SubDir):
        digests[relpath] = digest
        self._digest_path().write_text(json.dumps(digests))

    def _internal_path(self) -> Path:
        return self._path / ".internal"

    def is_internal(self) -> bool:
        return self._internal_path().is_file()

    def set_internal(self, internal: bool):
        path = self._internal_path()
        if internal:
            path.touch()
        elif path.is_file():
            path.unlink()

    def _dirty_path(self) -> Path:
        return self._path / ".dirty"

diff --git a/dlrepo/fs/tag.py b/dlrepo/fs/tag.py
index 6ca4f88085f4..793358f2437a 100644
--- a/dlrepo/fs/tag.py
+++ b/dlrepo/fs/tag.py
@@ -183,11 +183,6 @@ class Tag(SubDir):
            tasks.append(loop.create_task(_publish_file(file, digest)))
        await asyncio.gather(*tasks)

        if fmt.is_internal():
            LOG.debug("publishing internal format %s", fmt_url)
            async with semaphore:
                await sess.put(fmt_url, json={"artifact_format": {"internal": True}})

        # clear the dirty flag
        async with semaphore:
            await sess.patch(fmt_url)
diff --git a/dlrepo/templates/job.html b/dlrepo/templates/job.html
index fb9f27d23d69..e5ecc6719456 100644
--- a/dlrepo/templates/job.html
+++ b/dlrepo/templates/job.html
@@ -17,10 +17,7 @@
  <hr/>
  <div class="artifact-formats">
    {% for fmt in formats|sort(attribute="name") %}
    <div class="artifact-format{% if fmt.internal %} internal{% endif %}"
         {% if fmt.internal %}
         title="internal format (not for release)"
         {% endif %}>
    <div class="artifact-format">
      <a href="{{fmt.name}}" class="format">{{fmt.name}}/</a>
      <a href="{{fmt.name}}.tar" class="archive" title="Whole archive">
        {{fmt.name}}.tar
diff --git a/dlrepo/templates/product_version.html b/dlrepo/templates/product_version.html
index 65ecc868cc4b..9dc785181acd 100644
--- a/dlrepo/templates/product_version.html
+++ b/dlrepo/templates/product_version.html
@@ -14,10 +14,7 @@
  <hr/>
  <div class="artifact-formats">
    {% for fmt in version.artifact_formats|sort(attribute="name") %}
    <div class="artifact-format{% if fmt.internal %} internal{% endif %}"
         {% if fmt.internal %}
         title="internal format (not for release)"
         {% endif %}>
    <div class="artifact-format">
      <a href="{{fmt.name}}" class="format">{{fmt.name}}/</a>
      <a href="{{fmt.name}}.tar" class="archive" title="Whole archive">
        {{fmt.name}}.tar
diff --git a/dlrepo/views/fmt.py b/dlrepo/views/fmt.py
index bca70b61f458..fbd1418b2298 100644
--- a/dlrepo/views/fmt.py
+++ b/dlrepo/views/fmt.py
@@ -41,32 +41,12 @@ class FormatDirView(BaseView):
        data = {
            "artifact_format": {
                "name": fmt.name,
                "internal": fmt.is_internal(),
                "dirty": fmt.is_dirty(),
                "files": list(fmt.get_digests().keys()),
            },
        }
        return web.json_response(data)

    async def put(self):
        """
        Change the internal state for a format
        """
        version = self.request.match_info.get(
            "tag", self.request.match_info.get("version")
        )
        if "product" in self.request.match_info or version in ("latest", "stable"):
            raise web.HTTPMethodNotAllowed("PUT", ["GET"])
        fmt = _get_format(self.repo(), self.request.match_info)
        try:
            internal = (await self.json_body())["artifact_format"]["internal"]
            if not isinstance(internal, bool):
                raise TypeError()
        except (TypeError, KeyError) as e:
            raise web.HTTPBadRequest() from e
        fmt.set_internal(internal)
        return web.Response()

    async def patch(self):
        """
        Remove the dirty flag from a format.
diff --git a/dlrepo/views/job.py b/dlrepo/views/job.py
index 49fbbb3bc97f..79e5dddbe6ca 100644
--- a/dlrepo/views/job.py
+++ b/dlrepo/views/job.py
@@ -81,7 +81,6 @@ class JobView(JobArchiveView):
                    formats.append(
                        {
                            "name": f.name,
                            "internal": f.is_internal(),
                            "rpm": rpm,
                            "deb": deb,
                            "url": fmt_url,
diff --git a/dlrepo/views/product.py b/dlrepo/views/product.py
index 2d8381c3208b..9a0a11289bd2 100644
--- a/dlrepo/views/product.py
+++ b/dlrepo/views/product.py
@@ -203,7 +203,6 @@ class VersionView(BaseView):
                    formats.append(
                        {
                            "name": fmt.name,
                            "internal": fmt.is_internal(),
                            "rpm": rpm,
                            "deb": deb,
                            "url": fmt_url,
diff --git a/dlrepo/views/util.py b/dlrepo/views/util.py
index d6c92bc85cdf..b5404cc89825 100644
--- a/dlrepo/views/util.py
+++ b/dlrepo/views/util.py
@@ -67,7 +67,6 @@ class BaseView(web.View):
            "artifact_format": {
                "name": fmt.name,
                "relpath": relpath.rstrip("/"),
                "internal": fmt.is_internal(),
                "dirs": dirs,
                "files": files,
            },
diff --git a/docs/dlrepo-api.7.scdoc b/docs/dlrepo-api.7.scdoc
index 79db971036e3..0173ef9f0fdc 100644
--- a/docs/dlrepo-api.7.scdoc
+++ b/docs/dlrepo-api.7.scdoc
@@ -595,7 +595,6 @@ Get the list of files present in the specified _{format}_.
	{
		"artifact_format": {
			"name": "docs",
			"internal": false,
			"dirty": false,
			"files": [
				"config/index.html",
@@ -672,30 +671,6 @@ Check if the specified _{format}_ exists and is not _dirty_.
	- _404_: the specified _{format}_ does not exist or has the _dirty_ flag
	  set to _true_.

## PUT /branches/{branch}/{tag}/{job}/{format}/
## PUT /products/{product}/{variant}/{product_branch}/{version}/{format}/
## PUT /~{user}/branches/{branch}/{tag}/{job}/{format}/
## PUT /~{user}/products/{product}/{variant}/{product_branch}/{version}/{format}/

Set the _internal_ flag for the specified _{format}_. This flag is only
informative and does not imply any restrictions about the accessibility of the
format. Use *dlrepo-acls*(5) to restrict access to artifact formats.

*Access:*
	_rw_
*Request:*
	```
	{
		"artifact_format": {
			"internal": true
		}
	}
	```
*Errors:*
	- _400_: invalid parameters.
	- _404_: the specified _{tag}_ or _{version}_ does not exist.
	- _405_: _{tag}_ or _{version}_ is either _latest_ or _stable_.

## PATCH /branches/{branch}/{tag}/{job}/{format}/
## PATCH /products/{product}/{variant}/{product_branch}/{version}/{format}/
## PATCH /~{user}/branches/{branch}/{tag}/{job}/{format}/
diff --git a/docs/dlrepo-cli.1.scdoc b/docs/dlrepo-cli.1.scdoc
index dd383a1eba21..7df67b8a5b1f 100644
--- a/docs/dlrepo-cli.1.scdoc
+++ b/docs/dlrepo-cli.1.scdoc
@@ -186,23 +186,6 @@ _BRANCH_
_TAG_
	The tag name.

## dlrepo-cli internal [-u] BRANCH TAG JOB FORMAT

Set or unset the _internal_ status on an artifact format.

*-u*, *--unset*
	Unset the _internal_ status from the format instead of setting it.

_BRANCH_
	The branch name.
_TAG_
	The tag name.
_JOB_
	The job name.
_FORMAT_
	The artifact format name.


## dlrepo-cli delete [-f] BRANCH [TAG] [JOB]

Delete a job, a tag or a branch and all its tags recursively.
diff --git a/docs/dlrepo-layout.7.scdoc b/docs/dlrepo-layout.7.scdoc
index 2e8d993588a3..c26e58a1939c 100644
--- a/docs/dlrepo-layout.7.scdoc
+++ b/docs/dlrepo-layout.7.scdoc
@@ -149,9 +149,6 @@ _.dirty_
	file is exists, _HEAD_ requests to this artifact format will return
	a _404 Not found_ error (see *dlrepo-api*(7)). This file is deleted
	after a _PATCH_ request is made on this format.
_.internal_ (optional)
	Empty file created when the artifact format is marked as "internal".
	This flag is only informative.

## products/

diff --git a/scss/sections.scss b/scss/sections.scss
index 0c7ec16ffcd5..02002bd410c3 100644
--- a/scss/sections.scss
+++ b/scss/sections.scss
@@ -104,12 +104,6 @@ section.files {
    > .artifact-format:hover {
      background-color: lighten($lightgray, 5%);
    }
    > .artifact-format.internal {
      border-color: #fca5a5;
    }
    > .artifact-format.internal:hover {
      background-color: #fee2e2;
    }
  }

  > .whole-archive {
-- 
2.30.2

[PATCH dlrepo v2 3/7] job: add internal flag Export this patch

When releasing a tag, all of its jobs are published to the remote
server.

A tag may contain some jobs that are only used for internal testing and
that are never meant to be published to a remote server. Today though,
when a tag is released, all of its jobs are published.

To save space on the remote server, we add an "internal" job flag.
A job flagged "internal" is never published to a remote server.

Signed-off-by: Julien Floret <julien.floret@6wind.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>
---
Changes v1 -> v2:
 - we do not have to cope with "internal" flag on formats anymore
 - handle type errors in JobView.put()


 dlrepo-cli                 | 22 ++++++++++++++++++++++
 dlrepo/fs/job.py           | 14 ++++++++++++++
 dlrepo/fs/tag.py           |  2 ++
 dlrepo/views/job.py        | 17 ++++++++++++++---
 docs/dlrepo-api.7.scdoc    |  7 +++++--
 docs/dlrepo-cli.1.scdoc    | 15 +++++++++++++++
 docs/dlrepo-layout.7.scdoc |  2 ++
 7 files changed, 74 insertions(+), 5 deletions(-)

diff --git a/dlrepo-cli b/dlrepo-cli
index 351140ecabb0..c9a7e7f2391c 100755
--- a/dlrepo-cli
+++ b/dlrepo-cli
@@ -386,6 +386,28 @@ def release(args):
    client.put(url, {"tag": {"released": not args.unset}})


# --------------------------------------------------------------------------------------
@sub_command(
    Arg("branch", metavar="BRANCH", help="the branch name"),
    Arg("tag", metavar="TAG", help="the tag name"),
    Arg("job", metavar="JOB", help="the job name"),
    Arg(
        "-u",
        "--unset",
        action="store_true",
        help="unset the 'internal' status from the job instead of setting it",
    ),
)
def internal(args):
    """
    Set or unset the 'internal' status on a job. An internal job is never
    published to a remote server.
    """
    client = HttpClient(args.url)
    url = os.path.join("branches", args.branch, args.tag, args.job) + "/"
    client.put(url, {"job": {"internal": not args.unset}})


# --------------------------------------------------------------------------------------
@sub_command(
    Arg("branch", metavar="BRANCH", help="the branch name"),
diff --git a/dlrepo/fs/job.py b/dlrepo/fs/job.py
index 3ab3a698eb49..eb37e32c276f 100644
--- a/dlrepo/fs/job.py
+++ b/dlrepo/fs/job.py
@@ -6,6 +6,7 @@ import errno
import json
import logging
import os
from pathlib import Path
from typing import Dict, Iterator

from .fmt import ArtifactFormat
@@ -126,6 +127,19 @@ class Job(SubDir):
        elif path.is_file():
            path.unlink()

    def _internal_path(self) -> Path:
        return self._path / ".internal"

    def is_internal(self) -> bool:
        return self._internal_path().is_file()

    def set_internal(self, internal: bool):
        path = self._internal_path()
        if internal:
            path.touch()
        elif path.is_file():
            path.unlink()

    def add_metadata(self, new_data: Dict):
        if self.is_locked():
            raise FileExistsError("Job is locked")
diff --git a/dlrepo/fs/tag.py b/dlrepo/fs/tag.py
index 793358f2437a..33def160681e 100644
--- a/dlrepo/fs/tag.py
+++ b/dlrepo/fs/tag.py
@@ -132,6 +132,8 @@ class Tag(SubDir):
    async def _publish(self, sess: aiohttp.ClientSession, semaphore: asyncio.Semaphore):
        loop = asyncio.get_running_loop()
        for job in self.get_jobs():
            if job.is_internal():
                continue
            self._publish_status_path().write_text(f"uploading {job.name}\n")
            tasks = []
            for fmt in job.get_formats():
diff --git a/dlrepo/views/job.py b/dlrepo/views/job.py
index 79e5dddbe6ca..5297754ae6ee 100644
--- a/dlrepo/views/job.py
+++ b/dlrepo/views/job.py
@@ -67,6 +67,7 @@ class JobView(JobArchiveView):
            raise web.HTTPFound(job.url())
        html = "html" in self.request.headers.get("Accept", "json")
        data = {"job": job.get_metadata()}
        data["job"]["internal"] = job.is_internal()
        formats = []
        for f in job.get_formats():
            fmt_url = f.url()
@@ -94,16 +95,26 @@ class JobView(JobArchiveView):
        return web.json_response(data)

    async def put(self):
        job = self._get_job()
        try:
            locked = (await self.json_body())["job"]["locked"]
            if not isinstance(locked, bool):
            data = (await self.json_body())["job"]
            internal = data.get("internal")
            if internal is not None and not isinstance(internal, bool):
                raise TypeError()
            locked = data.get("locked")
            if locked is not None and not isinstance(locked, bool):
                raise TypeError()
        except (TypeError, KeyError) as e:
            raise web.HTTPBadRequest(reason="Invalid parameters") from e

        try:
            self._get_job().set_locked(locked)
            if internal is not None:
                job.set_internal(internal)
            if locked is not None:
                job.set_locked(locked)
        except FileNotFoundError as e:
            raise web.HTTPNotFound() from e

        return web.Response()

    async def patch(self):
diff --git a/docs/dlrepo-api.7.scdoc b/docs/dlrepo-api.7.scdoc
index 0173ef9f0fdc..018d6833822a 100644
--- a/docs/dlrepo-api.7.scdoc
+++ b/docs/dlrepo-api.7.scdoc
@@ -312,6 +312,7 @@ Get metadata and artifact formats for the specified _{job}_.
		"job": {
			"name": "foobaz-x86_64-postgresql",
			"locked": true,
			"internal": false,
			"product": "foobaz",
			"product_variant": "x86_64-postgresql",
			"product_branch": "3.2",
@@ -347,8 +348,9 @@ the archive contents after extraction.
## PUT /branches/{branch}/{tag}/{job}/
## PUT /~{user}/branches/{branch}/{tag}/{job}/

Lock or unlock the specified _{job}_. A locked job can no longer receive new
artifact uploads nor metadata change.
Change the internal and/or locked status of a _{job}_. An internal job can never
be published to another server. A locked job can no longer receive new artifact
uploads nor metadata change.

*Access:*
	_rw_
@@ -356,6 +358,7 @@ artifact uploads nor metadata change.
	```
	{
		"job": {
			"internal": false,
			"locked": true
		}
	}
diff --git a/docs/dlrepo-cli.1.scdoc b/docs/dlrepo-cli.1.scdoc
index 7df67b8a5b1f..db3f56495fb8 100644
--- a/docs/dlrepo-cli.1.scdoc
+++ b/docs/dlrepo-cli.1.scdoc
@@ -186,6 +186,21 @@ _BRANCH_
_TAG_
	The tag name.

## dlrepo-cli internal [-u] BRANCH TAG JOB

Set or unset the _internal_ status on a job. An internal job is never published
to a remote server.

*-u*, *--unset*
	Unset the _internal_ status instead of setting it.

_BRANCH_
	The branch name.
_TAG_
	The tag name.
_JOB_
	The job name.

## dlrepo-cli delete [-f] BRANCH [TAG] [JOB]

Delete a job, a tag or a branch and all its tags recursively.
diff --git a/docs/dlrepo-layout.7.scdoc b/docs/dlrepo-layout.7.scdoc
index c26e58a1939c..7fcd2ef49b1f 100644
--- a/docs/dlrepo-layout.7.scdoc
+++ b/docs/dlrepo-layout.7.scdoc
@@ -120,6 +120,8 @@ _.job_ (optional)
	Symbolic link to a _{version}_ in the _products/_ tree. This is only
	present if the _product_, _version_, _product_branch_ and
	_product_variant_ metadata have been set for this job.
_.internal_ (optional)
	Empty file created when the job is marked as "internal".

## branches/{branch}/{tag}/{job}/{format}/

-- 
2.30.2

[PATCH dlrepo v2 5/7] job: display upload time Export this patch

Add the job upload time (as an integer timestamp) to the job metadata.
Its value is determined by looking at the modification time of a
.stamp file, which is created when the job is uploaded.

The upload time is then be displayed on the job web page, in
a user-friendly format indicating the user's local time.

Signed-off-by: Julien Floret <julien.floret@6wind.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>
---
Changes v1 -> v2:
 - A numeric timestamp is now exported in the job metadata. The
   conversion into a formatted string is now made
   in the html template.
 - Added some missing job.create() calls
 
 dlrepo/fs/container.py     |  1 +
 dlrepo/fs/fmt.py           |  2 +-
 dlrepo/fs/job.py           | 21 ++++++++++++
 dlrepo/templates/job.html  |  5 +++
 dlrepo/views/__init__.py   | 11 ++++++-
 dlrepo/views/job.py        |  1 +
 dlrepo/views/localtime.py  | 66 ++++++++++++++++++++++++++++++++++++++
 docs/dlrepo-api.7.scdoc    |  1 +
 docs/dlrepo-layout.7.scdoc |  3 ++
 9 files changed, 109 insertions(+), 2 deletions(-)
 create mode 100644 dlrepo/views/localtime.py

diff --git a/dlrepo/fs/container.py b/dlrepo/fs/container.py
index 86c38baffd08..4a6dde01e6d9 100644
--- a/dlrepo/fs/container.py
+++ b/dlrepo/fs/container.py
@@ -99,6 +99,7 @@ class ContainerRegistry:
        # manifest, we can read their size on the filesystem.

        # link config blob into the job folder
        job.create()
        config = manifest.get("config", {})
        config["size"] = self._link_blob_to_job(config.get("digest"), job)

diff --git a/dlrepo/fs/fmt.py b/dlrepo/fs/fmt.py
index 29c372e0627e..6d67ffbeb331 100644
--- a/dlrepo/fs/fmt.py
+++ b/dlrepo/fs/fmt.py
@@ -133,7 +133,7 @@ class ArtifactFormat(SubDir):
    def set_dirty(self, dirty: bool):
        path = self._dirty_path()
        if dirty:
            path.parent.mkdir(mode=0o755, parents=True, exist_ok=True)
            self.create()
            path.touch()
        elif path.is_file():
            path.unlink()
diff --git a/dlrepo/fs/job.py b/dlrepo/fs/job.py
index eb37e32c276f..5120f159f5f9 100644
--- a/dlrepo/fs/job.py
+++ b/dlrepo/fs/job.py
@@ -23,6 +23,26 @@ class Job(SubDir):
    TODO
    """

    def create(self):
        super().create()
        stamp = self._path / ".stamp"
        if not stamp.exists():
            stamp.touch()

    @classmethod
    def creation_date(cls, j):
        stamp = j.path() / ".stamp"
        if stamp.is_file():
            # prefer mtime over ctime
            # on UNIX, ctime is "the time of most recent metadata change" whereas
            # mtime is "most recent content modification"
            return stamp.stat().st_mtime
        return 0

    @property
    def timestamp(self) -> int:
        return Job.creation_date(self)

    def get_formats(self) -> Iterator[ArtifactFormat]:
        yield from ArtifactFormat.all(self)

@@ -121,6 +141,7 @@ class Job(SubDir):
    def set_locked(self, locked: bool):
        if not self.exists():
            raise FileNotFoundError()
        self.create()
        path = self._lock_path()
        if locked:
            path.touch()
diff --git a/dlrepo/templates/job.html b/dlrepo/templates/job.html
index e5ecc6719456..bd6ca42bb17d 100644
--- a/dlrepo/templates/job.html
+++ b/dlrepo/templates/job.html
@@ -6,6 +6,7 @@

{% set job_name = job.pop("name") %}
{% set formats = job.pop("artifact_formats", []) %}
{% set timestamp = job.pop("timestamp") %}

{% block breadcrumbs %}
{{breadcrumbs(['branches', branch, tag, job_name])}}
@@ -47,6 +48,10 @@
    <span class="value">{{value}}</span>
  </div>
  {% endfor %}
  <div class="metadata">
    <span class="key">upload_time</span>
    <span class="value">{{timestamp|local_time}}</span>
  </div>
</section>

{% if job.product and job.product_variant and job.product_branch and job.version %}
diff --git a/dlrepo/views/__init__.py b/dlrepo/views/__init__.py
index 85b5b7b44321..1eb76975117c 100644
--- a/dlrepo/views/__init__.py
+++ b/dlrepo/views/__init__.py
@@ -3,7 +3,7 @@
# SPDX-License-Identifier: BSD-3-Clause

import asyncio
from datetime import datetime
from datetime import datetime, timezone
import os
import pathlib

@@ -11,6 +11,7 @@ from aiohttp import web
import aiohttp_jinja2
import jinja2

from . import localtime
from ..fs.util import human_readable
from .artifact import ArtifactView
from .branch import BranchesView, BranchView
@@ -170,6 +171,13 @@ async def template_vars(request):

# --------------------------------------------------------------------------------------
def add_routes(app):
    def pretty_localtime(timestamp, fmt="%Y-%m-%d %H:%M:%S UTC%z"):
        if not timestamp:
            return "n/a"
        utc_time = datetime.fromtimestamp(timestamp, timezone.utc)
        local_time = utc_time.astimezone(localtime.LOCAL)
        return local_time.strftime(fmt)

    template_dirs = []
    if os.getenv("DLREPO_TEMPLATES_DIR"):
        template_dirs.append(os.getenv("DLREPO_TEMPLATES_DIR"))
@@ -188,6 +196,7 @@ def add_routes(app):
        extensions=["jinja2.ext.do"],
        trim_blocks=True,
        lstrip_blocks=True,
        filters={"local_time": pretty_localtime},
    )
    for route in (
        HomeView,
diff --git a/dlrepo/views/job.py b/dlrepo/views/job.py
index 5297754ae6ee..9ad32e43b462 100644
--- a/dlrepo/views/job.py
+++ b/dlrepo/views/job.py
@@ -68,6 +68,7 @@ class JobView(JobArchiveView):
        html = "html" in self.request.headers.get("Accept", "json")
        data = {"job": job.get_metadata()}
        data["job"]["internal"] = job.is_internal()
        data["job"]["timestamp"] = job.timestamp
        formats = []
        for f in job.get_formats():
            fmt_url = f.url()
diff --git a/dlrepo/views/localtime.py b/dlrepo/views/localtime.py
new file mode 100644
index 000000000000..f1ad770d6521
--- /dev/null
+++ b/dlrepo/views/localtime.py
@@ -0,0 +1,66 @@
# Code imported from python official documentation
# https://docs.python.org/3.7/library/datetime.html

from datetime import datetime, timedelta, tzinfo
import time as _time


ZERO = timedelta(0)
HOUR = timedelta(hours=1)
SECOND = timedelta(seconds=1)

# A class capturing the platform's idea of local time.
# (May result in wrong values on historical times in
#  timezones where UTC offset and/or the DST rules had
#  changed in the past.)

STDOFFSET = timedelta(seconds=-_time.timezone)
if _time.daylight:
    DSTOFFSET = timedelta(seconds=-_time.altzone)
else:
    DSTOFFSET = STDOFFSET

DSTDIFF = DSTOFFSET - STDOFFSET


class LocalTimezone(tzinfo):
    def fromutc(self, dt):
        assert dt.tzinfo is self
        stamp = (dt - datetime(1970, 1, 1, tzinfo=self)) // SECOND
        args = _time.localtime(stamp)[:6]
        dst_diff = DSTDIFF // SECOND
        # Detect fold
        fold = args == _time.localtime(stamp - dst_diff)
        return datetime(*args, microsecond=dt.microsecond, tzinfo=self, fold=fold)

    def utcoffset(self, dt):
        if self._isdst(dt):
            return DSTOFFSET
        return STDOFFSET

    def dst(self, dt):
        if self._isdst(dt):
            return DSTDIFF
        return ZERO

    def tzname(self, dt):
        return _time.tzname[self._isdst(dt)]

    def _isdst(self, dt):
        tt = (
            dt.year,
            dt.month,
            dt.day,
            dt.hour,
            dt.minute,
            dt.second,
            dt.weekday(),
            0,
            0,
        )
        stamp = _time.mktime(tt)
        tt = _time.localtime(stamp)
        return tt.tm_isdst > 0


LOCAL = LocalTimezone()
diff --git a/docs/dlrepo-api.7.scdoc b/docs/dlrepo-api.7.scdoc
index 018d6833822a..4e75f883e61b 100644
--- a/docs/dlrepo-api.7.scdoc
+++ b/docs/dlrepo-api.7.scdoc
@@ -313,6 +313,7 @@ Get metadata and artifact formats for the specified _{job}_.
			"name": "foobaz-x86_64-postgresql",
			"locked": true,
			"internal": false,
			"timestamp": 1637001613.5863628,
			"product": "foobaz",
			"product_variant": "x86_64-postgresql",
			"product_branch": "3.2",
diff --git a/docs/dlrepo-layout.7.scdoc b/docs/dlrepo-layout.7.scdoc
index 7fcd2ef49b1f..49ab60062105 100644
--- a/docs/dlrepo-layout.7.scdoc
+++ b/docs/dlrepo-layout.7.scdoc
@@ -112,6 +112,9 @@ _.metadata_ (optional)
		"baz": "bar"
	}
	```
_.stamp_
	Empty file created when the job is first created. This is used to
	display the job creation time.
_.locked_ (optional)
	Empty file created when the job is locked to avoid further
	modifications. A locked job cannot receive file uploads nor metadata
-- 
2.30.2

[PATCH dlrepo v2 6/7] tag: clean published jobs before re-publishing Export this patch

Sometimes, after a tag has been published to a remote server, we need to
rebuild the local tag and publish it again, because something was wrong
in the original build. The caveat is that when re-publishing a job, any
file in the original job that does not exist anymore in the updated job
will not be removed from the remote server. This is especially
problematic when re-publishing a container: since the manifest file is
named after its sha256 sum, we end up with two manifest files on the
remote server and docker access is denied.

Force deletion of jobs already published before re-publishing them.

Signed-off-by: Julien Floret <julien.floret@6wind.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>
---
This one is a new patch.

 dlrepo/fs/tag.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/dlrepo/fs/tag.py b/dlrepo/fs/tag.py
index 33def160681e..40a0bfcf6817 100644
--- a/dlrepo/fs/tag.py
+++ b/dlrepo/fs/tag.py
@@ -134,6 +134,9 @@ class Tag(SubDir):
        for job in self.get_jobs():
            if job.is_internal():
                continue
            job_url = job.url()
            # in case job is already present on the server, delete it before re-uploading it
            await sess.delete(job_url, params={"force": "true"}, raise_for_status=False)
            self._publish_status_path().write_text(f"uploading {job.name}\n")
            tasks = []
            for fmt in job.get_formats():
@@ -142,7 +145,6 @@ class Tag(SubDir):
            metadata = job.get_metadata()
            del metadata["name"]
            del metadata["locked"]
            job_url = job.url()
            LOG.debug("publishing job metadata %s", job_url)
            async with semaphore:
                await sess.patch(job_url, json={"job": metadata})
-- 
2.30.2

[PATCH dlrepo v2 7/7] tag: robustify TagView.put() Export this patch

Check the types of the input parameters: when at least one has not the
right type, respond with HTTPBadRequest and apply none (previously,
parameters of the wrong type were silently ignored.)
Also catch the FileNotFoundError exception to respond a proper
HTTPNotFound.

Signed-off-by: Julien Floret <julien.floret@6wind.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>
---
This one is a new patch, prompted by the added type error checks in
patch 3.

 dlrepo/views/tag.py | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)

diff --git a/dlrepo/views/tag.py b/dlrepo/views/tag.py
index 7430269f1e2e..2d20bfe6f9a8 100644
--- a/dlrepo/views/tag.py
+++ b/dlrepo/views/tag.py
@@ -73,13 +73,24 @@ class TagView(BaseView):
        tag = self._get_tag()
        try:
            data = (await self.json_body())["tag"]
            released = data.get("released")
            if released is not None and not isinstance(released, bool):
                raise TypeError()
            locked = data.get("locked")
            if locked is not None and not isinstance(locked, bool):
                raise TypeError()
        except (TypeError, KeyError) as e:
            raise web.HTTPBadRequest(reason="invalid parameters") from e
        if isinstance(data.get("released"), bool):
            semaphore = self.request.app["dlrepo_publish_semaphore"]
            tag.set_released(data["released"], semaphore)
        if isinstance(data.get("locked"), bool):
            tag.set_locked(data["locked"])

        try:
            if released is not None:
                semaphore = self.request.app["dlrepo_publish_semaphore"]
                tag.set_released(released, semaphore)
            if locked is not None:
                tag.set_locked(locked)
        except FileNotFoundError as e:
            raise web.HTTPNotFound() from e

        return web.Response()

    async def delete(self):
-- 
2.30.2