~rjarry/dlrepo

8 2

[PATCH dlrepo v3 0/7] various job improvements

Details
Message ID
<20220516101638.10977-1-julien.floret@6wind.com>
DKIM signature
missing
Download raw message
This series is motivated 
This v3 is sent because the git server did not recognized the v2 because
of an unfortunate in-reply-to. Otherwise, no change since v2. 

- the first patch does some cleanup in dlrepo-cli.
- patches 2 to 4 add an "internal" flag to the job to prevent jobs meant
  for internal testing to be published to an external server. The
  "internal" flag on formats was confusing, it is dropped.
- patch 5 exports the job upload timestamp in the metadata, to display
  it on the html page.
- patch 6 ensures that when re-publishing a job it does not keep stale
  files from the previous publishing
- patch 7 adds some checks to TagView.put(), inspired by changes done in
  JobView.put() in patch 3.

Julien Floret (7):
  cli: remove unused local_dir() function
  fmt: remove internal flag
  job: add internal flag
  tests: check internal jobs are not published
  job: display upload time
  tag: clean published jobs before re-publishing
  tag: robustify TagView.put()

 dlrepo-cli                                    | 17 ++---
 dlrepo/fs/container.py                        |  1 +
 dlrepo/fs/fmt.py                              | 19 +-----
 dlrepo/fs/job.py                              | 35 ++++++++++
 dlrepo/fs/tag.py                              | 11 ++--
 dlrepo/templates/job.html                     | 10 +--
 dlrepo/templates/product_version.html         |  5 +-
 dlrepo/views/__init__.py                      | 11 +++-
 dlrepo/views/fmt.py                           | 20 ------
 dlrepo/views/job.py                           | 19 ++++--
 dlrepo/views/localtime.py                     | 66 +++++++++++++++++++
 dlrepo/views/product.py                       |  1 -
 dlrepo/views/tag.py                           | 21 ++++--
 dlrepo/views/util.py                          |  1 -
 docs/dlrepo-api.7.scdoc                       | 33 ++--------
 docs/dlrepo-cli.1.scdoc                       | 10 ++-
 docs/dlrepo-layout.7.scdoc                    |  8 ++-
 scss/sections.scss                            |  6 --
 .../branch/tag/internal_job/.internal         |  0
 .../branch/tag/internal_job/fmt1/.digests     |  1 +
 .../branches/branch/tag/internal_job/fmt1/a   |  1 +
 .../branches/branch/tag/internal_job/fmt1/b   |  1 +
 tests/test_publish.py                         |  5 ++
 23 files changed, 185 insertions(+), 117 deletions(-)
 create mode 100644 dlrepo/views/localtime.py
 create mode 100644 tests/publish_reference/branches/branch/tag/internal_job/.internal
 create mode 100644 tests/publish_reference/branches/branch/tag/internal_job/fmt1/.digests
 create mode 100644 tests/publish_reference/branches/branch/tag/internal_job/fmt1/a
 create mode 100644 tests/publish_reference/branches/branch/tag/internal_job/fmt1/b

-- 
2.30.2

[PATCH dlrepo v3 1/7] cli: remove unused local_dir() function

Details
Message ID
<20220516101638.10977-2-julien.floret@6wind.com>
In-Reply-To
<20220516101638.10977-1-julien.floret@6wind.com> (view parent)
DKIM signature
missing
Download raw message
Patch: +0 -7
This function is present since the import of the file but has never been
used.

Fixes: 49ff6a714f43 ("Add client script")
Signed-off-by: Julien Floret <julien.floret@6wind.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>
---
 dlrepo-cli | 7 -------
 1 file changed, 7 deletions(-)

diff --git a/dlrepo-cli b/dlrepo-cli
index 298387df9631..476d3350ad29 100755
--- a/dlrepo-cli
+++ b/dlrepo-cli
@@ -131,13 +131,6 @@ def local_path(value):
    return value


# --------------------------------------------------------------------------------------
def local_dir(value):
    if not os.path.isdir(value):
        raise argparse.ArgumentTypeError("%r: No such directory" % (value,))
    return value


# --------------------------------------------------------------------------------------
def job_param(value):
    match = re.match(r"^(\w+)=(.*)$", value)
-- 
2.30.2

[PATCH dlrepo v3 3/7] job: add internal flag

Details
Message ID
<20220516101638.10977-4-julien.floret@6wind.com>
In-Reply-To
<20220516101638.10977-1-julien.floret@6wind.com> (view parent)
DKIM signature
missing
Download raw message
Patch: +74 -5
When releasing a tag, all of its jobs are published to the remote
server.

A tag may contain some jobs that are only used for internal testing and
that are never meant to be published to a remote server. Today though,
when a tag is released, all of its jobs are published.

To save space on the remote server, we add an "internal" job flag.
A job flagged "internal" is never published to a remote server.

Signed-off-by: Julien Floret <julien.floret@6wind.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>
---
 dlrepo-cli                 | 22 ++++++++++++++++++++++
 dlrepo/fs/job.py           | 14 ++++++++++++++
 dlrepo/fs/tag.py           |  2 ++
 dlrepo/views/job.py        | 17 ++++++++++++++---
 docs/dlrepo-api.7.scdoc    |  7 +++++--
 docs/dlrepo-cli.1.scdoc    | 15 +++++++++++++++
 docs/dlrepo-layout.7.scdoc |  2 ++
 7 files changed, 74 insertions(+), 5 deletions(-)

diff --git a/dlrepo-cli b/dlrepo-cli
index 351140ecabb0..c9a7e7f2391c 100755
--- a/dlrepo-cli
+++ b/dlrepo-cli
@@ -386,6 +386,28 @@ def release(args):
    client.put(url, {"tag": {"released": not args.unset}})


# --------------------------------------------------------------------------------------
@sub_command(
    Arg("branch", metavar="BRANCH", help="the branch name"),
    Arg("tag", metavar="TAG", help="the tag name"),
    Arg("job", metavar="JOB", help="the job name"),
    Arg(
        "-u",
        "--unset",
        action="store_true",
        help="unset the 'internal' status from the job instead of setting it",
    ),
)
def internal(args):
    """
    Set or unset the 'internal' status on a job. An internal job is never
    published to a remote server.
    """
    client = HttpClient(args.url)
    url = os.path.join("branches", args.branch, args.tag, args.job) + "/"
    client.put(url, {"job": {"internal": not args.unset}})


# --------------------------------------------------------------------------------------
@sub_command(
    Arg("branch", metavar="BRANCH", help="the branch name"),
diff --git a/dlrepo/fs/job.py b/dlrepo/fs/job.py
index 3ab3a698eb49..eb37e32c276f 100644
--- a/dlrepo/fs/job.py
+++ b/dlrepo/fs/job.py
@@ -6,6 +6,7 @@ import errno
import json
import logging
import os
from pathlib import Path
from typing import Dict, Iterator

from .fmt import ArtifactFormat
@@ -126,6 +127,19 @@ class Job(SubDir):
        elif path.is_file():
            path.unlink()

    def _internal_path(self) -> Path:
        return self._path / ".internal"

    def is_internal(self) -> bool:
        return self._internal_path().is_file()

    def set_internal(self, internal: bool):
        path = self._internal_path()
        if internal:
            path.touch()
        elif path.is_file():
            path.unlink()

    def add_metadata(self, new_data: Dict):
        if self.is_locked():
            raise FileExistsError("Job is locked")
diff --git a/dlrepo/fs/tag.py b/dlrepo/fs/tag.py
index 793358f2437a..33def160681e 100644
--- a/dlrepo/fs/tag.py
+++ b/dlrepo/fs/tag.py
@@ -132,6 +132,8 @@ class Tag(SubDir):
    async def _publish(self, sess: aiohttp.ClientSession, semaphore: asyncio.Semaphore):
        loop = asyncio.get_running_loop()
        for job in self.get_jobs():
            if job.is_internal():
                continue
            self._publish_status_path().write_text(f"uploading {job.name}\n")
            tasks = []
            for fmt in job.get_formats():
diff --git a/dlrepo/views/job.py b/dlrepo/views/job.py
index 79e5dddbe6ca..5297754ae6ee 100644
--- a/dlrepo/views/job.py
+++ b/dlrepo/views/job.py
@@ -67,6 +67,7 @@ class JobView(JobArchiveView):
            raise web.HTTPFound(job.url())
        html = "html" in self.request.headers.get("Accept", "json")
        data = {"job": job.get_metadata()}
        data["job"]["internal"] = job.is_internal()
        formats = []
        for f in job.get_formats():
            fmt_url = f.url()
@@ -94,16 +95,26 @@ class JobView(JobArchiveView):
        return web.json_response(data)

    async def put(self):
        job = self._get_job()
        try:
            locked = (await self.json_body())["job"]["locked"]
            if not isinstance(locked, bool):
            data = (await self.json_body())["job"]
            internal = data.get("internal")
            if internal is not None and not isinstance(internal, bool):
                raise TypeError()
            locked = data.get("locked")
            if locked is not None and not isinstance(locked, bool):
                raise TypeError()
        except (TypeError, KeyError) as e:
            raise web.HTTPBadRequest(reason="Invalid parameters") from e

        try:
            self._get_job().set_locked(locked)
            if internal is not None:
                job.set_internal(internal)
            if locked is not None:
                job.set_locked(locked)
        except FileNotFoundError as e:
            raise web.HTTPNotFound() from e

        return web.Response()

    async def patch(self):
diff --git a/docs/dlrepo-api.7.scdoc b/docs/dlrepo-api.7.scdoc
index 0173ef9f0fdc..018d6833822a 100644
--- a/docs/dlrepo-api.7.scdoc
+++ b/docs/dlrepo-api.7.scdoc
@@ -312,6 +312,7 @@ Get metadata and artifact formats for the specified _{job}_.
		"job": {
			"name": "foobaz-x86_64-postgresql",
			"locked": true,
			"internal": false,
			"product": "foobaz",
			"product_variant": "x86_64-postgresql",
			"product_branch": "3.2",
@@ -347,8 +348,9 @@ the archive contents after extraction.
## PUT /branches/{branch}/{tag}/{job}/
## PUT /~{user}/branches/{branch}/{tag}/{job}/

Lock or unlock the specified _{job}_. A locked job can no longer receive new
artifact uploads nor metadata change.
Change the internal and/or locked status of a _{job}_. An internal job can never
be published to another server. A locked job can no longer receive new artifact
uploads nor metadata change.

*Access:*
	_rw_
@@ -356,6 +358,7 @@ artifact uploads nor metadata change.
	```
	{
		"job": {
			"internal": false,
			"locked": true
		}
	}
diff --git a/docs/dlrepo-cli.1.scdoc b/docs/dlrepo-cli.1.scdoc
index 7df67b8a5b1f..db3f56495fb8 100644
--- a/docs/dlrepo-cli.1.scdoc
+++ b/docs/dlrepo-cli.1.scdoc
@@ -186,6 +186,21 @@ _BRANCH_
_TAG_
	The tag name.

## dlrepo-cli internal [-u] BRANCH TAG JOB

Set or unset the _internal_ status on a job. An internal job is never published
to a remote server.

*-u*, *--unset*
	Unset the _internal_ status instead of setting it.

_BRANCH_
	The branch name.
_TAG_
	The tag name.
_JOB_
	The job name.

## dlrepo-cli delete [-f] BRANCH [TAG] [JOB]

Delete a job, a tag or a branch and all its tags recursively.
diff --git a/docs/dlrepo-layout.7.scdoc b/docs/dlrepo-layout.7.scdoc
index c26e58a1939c..7fcd2ef49b1f 100644
--- a/docs/dlrepo-layout.7.scdoc
+++ b/docs/dlrepo-layout.7.scdoc
@@ -120,6 +120,8 @@ _.job_ (optional)
	Symbolic link to a _{version}_ in the _products/_ tree. This is only
	present if the _product_, _version_, _product_branch_ and
	_product_variant_ metadata have been set for this job.
_.internal_ (optional)
	Empty file created when the job is marked as "internal".

## branches/{branch}/{tag}/{job}/{format}/

-- 
2.30.2

[PATCH dlrepo v3 4/7] tests: check internal jobs are not published

Details
Message ID
<20220516101638.10977-5-julien.floret@6wind.com>
In-Reply-To
<20220516101638.10977-1-julien.floret@6wind.com> (view parent)
DKIM signature
missing
Download raw message
The "internal_job" job is flagged "internal". Test that it is not
published to the remote server.

Signed-off-by: Julien Floret <julien.floret@6wind.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>
---
 .../branches/branch/tag/internal_job/.internal               | 0
 .../branches/branch/tag/internal_job/fmt1/.digests           | 1 +
 .../branches/branch/tag/internal_job/fmt1/a                  | 1 +
 .../branches/branch/tag/internal_job/fmt1/b                  | 1 +
 tests/test_publish.py                                        | 5 +++++
 5 files changed, 8 insertions(+)
 create mode 100644 tests/publish_reference/branches/branch/tag/internal_job/.internal
 create mode 100644 tests/publish_reference/branches/branch/tag/internal_job/fmt1/.digests
 create mode 100644 tests/publish_reference/branches/branch/tag/internal_job/fmt1/a
 create mode 100644 tests/publish_reference/branches/branch/tag/internal_job/fmt1/b

diff --git a/tests/publish_reference/branches/branch/tag/internal_job/.internal b/tests/publish_reference/branches/branch/tag/internal_job/.internal
new file mode 100644
index 000000000000..e69de29bb2d1
diff --git a/tests/publish_reference/branches/branch/tag/internal_job/fmt1/.digests b/tests/publish_reference/branches/branch/tag/internal_job/fmt1/.digests
new file mode 100644
index 000000000000..7ae13f34b9e3
--- /dev/null
+++ b/tests/publish_reference/branches/branch/tag/internal_job/fmt1/.digests
@@ -0,0 +1 @@
+{"b": "sha256:0263829989b6fd954f72baaf2fc64bc2e2f01d692d4de72986ea808f6e99813f", "a": "sha256:87428fc522803d31065e7bce3cf03fe475096631e5e07bbd7a0fde60c4cf25c7"}
\ No newline at end of file
diff --git a/tests/publish_reference/branches/branch/tag/internal_job/fmt1/a b/tests/publish_reference/branches/branch/tag/internal_job/fmt1/a
new file mode 100644
index 000000000000..78981922613b
--- /dev/null
+++ b/tests/publish_reference/branches/branch/tag/internal_job/fmt1/a
@@ -0,0 +1 @@
+a
diff --git a/tests/publish_reference/branches/branch/tag/internal_job/fmt1/b b/tests/publish_reference/branches/branch/tag/internal_job/fmt1/b
new file mode 100644
index 000000000000..61780798228d
--- /dev/null
+++ b/tests/publish_reference/branches/branch/tag/internal_job/fmt1/b
@@ -0,0 +1 @@
+b
diff --git a/tests/test_publish.py b/tests/test_publish.py
index cceb07c97695..7c9f2f572749 100644
--- a/tests/test_publish.py
+++ b/tests/test_publish.py
@@ -83,3 +83,8 @@ async def test_publish(dlrepo_servers):  # pylint: disable=redefined-outer-name
                     assert resp.status == 200
                     data_pub = await resp.text()
                     assert data_pub == data
+            sha_url = "/branches/branch/tag/internal_job/fmt1.sha256"
+            resp = await sess.get(sha_url)
+            assert resp.status == 200
+            resp = await pub_sess.get(sha_url)
+            assert resp.status == 404
-- 
2.30.2

[PATCH dlrepo v3 2/7] fmt: remove internal flag

Details
Message ID
<20220516101638.10977-3-julien.floret@6wind.com>
In-Reply-To
<20220516101638.10977-1-julien.floret@6wind.com> (view parent)
DKIM signature
missing
Download raw message
Patch: +3 -125
The "internal" flag on artifacts formats can be quite misleading,
because it is only informative and does not prevent the format to be
uploaded to a remote publish server. ACLs should be more than enough to
hide some formats if need be.

Note: after upgrading your server, you must remove the ".internal"
files manually from your server artifact database:

  # rm -f $DLREPO_ROOT_DIR/branches/*/*/*/*/.internal

Signed-off-by: Julien Floret <julien.floret@6wind.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>
---
 dlrepo-cli                            | 22 ----------------------
 dlrepo/fs/fmt.py                      | 17 +----------------
 dlrepo/fs/tag.py                      |  5 -----
 dlrepo/templates/job.html             |  5 +----
 dlrepo/templates/product_version.html |  5 +----
 dlrepo/views/fmt.py                   | 20 --------------------
 dlrepo/views/job.py                   |  1 -
 dlrepo/views/product.py               |  1 -
 dlrepo/views/util.py                  |  1 -
 docs/dlrepo-api.7.scdoc               | 25 -------------------------
 docs/dlrepo-cli.1.scdoc               | 17 -----------------
 docs/dlrepo-layout.7.scdoc            |  3 ---
 scss/sections.scss                    |  6 ------
 13 files changed, 3 insertions(+), 125 deletions(-)

diff --git a/dlrepo-cli b/dlrepo-cli
index 476d3350ad29..351140ecabb0 100755
--- a/dlrepo-cli
+++ b/dlrepo-cli
@@ -386,28 +386,6 @@ def release(args):
    client.put(url, {"tag": {"released": not args.unset}})


# --------------------------------------------------------------------------------------
@sub_command(
    Arg("branch", metavar="BRANCH", help="the branch name"),
    Arg("tag", metavar="TAG", help="the tag name"),
    Arg("job", metavar="JOB", help="the job name"),
    Arg("format", metavar="FORMAT", help="the artifact format"),
    Arg(
        "-u",
        "--unset",
        action="store_true",
        help="unset the 'internal' status from the format instead of setting it",
    ),
)
def internal(args):
    """
    Set or unset the 'internal' status on an artifact format.
    """
    client = HttpClient(args.url)
    url = os.path.join("branches", args.branch, args.tag, args.job, args.format) + "/"
    client.put(url, {"artifact_format": {"internal": not args.unset}})


# --------------------------------------------------------------------------------------
@sub_command(
    Arg("branch", metavar="BRANCH", help="the branch name"),
diff --git a/dlrepo/fs/fmt.py b/dlrepo/fs/fmt.py
index a58602198d0b..29c372e0627e 100644
--- a/dlrepo/fs/fmt.py
+++ b/dlrepo/fs/fmt.py
@@ -44,14 +44,12 @@ class ArtifactFormat(SubDir):

    @cachedmethod(lambda self: self._is_reserved_cache)
    def _is_reserved_file(self, path, *, resolve=False):
        internal = self._internal_path()
        digests = self._digest_path()
        dirty = self._dirty_path()
        if resolve:
            internal = internal.resolve()
            digests = digests.resolve()
            dirty = dirty.resolve()
        return path in (internal, digests, dirty)
        return path in (digests, dirty)

    def list_dir(self, relpath: str) -> Tuple[List[str], List[str]]:
        path = self.get_filepath(relpath)
@@ -126,19 +124,6 @@ class ArtifactFormat(SubDir):
        digests[relpath] = digest
        self._digest_path().write_text(json.dumps(digests))

    def _internal_path(self) -> Path:
        return self._path / ".internal"

    def is_internal(self) -> bool:
        return self._internal_path().is_file()

    def set_internal(self, internal: bool):
        path = self._internal_path()
        if internal:
            path.touch()
        elif path.is_file():
            path.unlink()

    def _dirty_path(self) -> Path:
        return self._path / ".dirty"

diff --git a/dlrepo/fs/tag.py b/dlrepo/fs/tag.py
index 6ca4f88085f4..793358f2437a 100644
--- a/dlrepo/fs/tag.py
+++ b/dlrepo/fs/tag.py
@@ -183,11 +183,6 @@ class Tag(SubDir):
            tasks.append(loop.create_task(_publish_file(file, digest)))
        await asyncio.gather(*tasks)

        if fmt.is_internal():
            LOG.debug("publishing internal format %s", fmt_url)
            async with semaphore:
                await sess.put(fmt_url, json={"artifact_format": {"internal": True}})

        # clear the dirty flag
        async with semaphore:
            await sess.patch(fmt_url)
diff --git a/dlrepo/templates/job.html b/dlrepo/templates/job.html
index fb9f27d23d69..e5ecc6719456 100644
--- a/dlrepo/templates/job.html
+++ b/dlrepo/templates/job.html
@@ -17,10 +17,7 @@
  <hr/>
  <div class="artifact-formats">
    {% for fmt in formats|sort(attribute="name") %}
    <div class="artifact-format{% if fmt.internal %} internal{% endif %}"
         {% if fmt.internal %}
         title="internal format (not for release)"
         {% endif %}>
    <div class="artifact-format">
      <a href="{{fmt.name}}" class="format">{{fmt.name}}/</a>
      <a href="{{fmt.name}}.tar" class="archive" title="Whole archive">
        {{fmt.name}}.tar
diff --git a/dlrepo/templates/product_version.html b/dlrepo/templates/product_version.html
index 65ecc868cc4b..9dc785181acd 100644
--- a/dlrepo/templates/product_version.html
+++ b/dlrepo/templates/product_version.html
@@ -14,10 +14,7 @@
  <hr/>
  <div class="artifact-formats">
    {% for fmt in version.artifact_formats|sort(attribute="name") %}
    <div class="artifact-format{% if fmt.internal %} internal{% endif %}"
         {% if fmt.internal %}
         title="internal format (not for release)"
         {% endif %}>
    <div class="artifact-format">
      <a href="{{fmt.name}}" class="format">{{fmt.name}}/</a>
      <a href="{{fmt.name}}.tar" class="archive" title="Whole archive">
        {{fmt.name}}.tar
diff --git a/dlrepo/views/fmt.py b/dlrepo/views/fmt.py
index bca70b61f458..fbd1418b2298 100644
--- a/dlrepo/views/fmt.py
+++ b/dlrepo/views/fmt.py
@@ -41,32 +41,12 @@ class FormatDirView(BaseView):
        data = {
            "artifact_format": {
                "name": fmt.name,
                "internal": fmt.is_internal(),
                "dirty": fmt.is_dirty(),
                "files": list(fmt.get_digests().keys()),
            },
        }
        return web.json_response(data)

    async def put(self):
        """
        Change the internal state for a format
        """
        version = self.request.match_info.get(
            "tag", self.request.match_info.get("version")
        )
        if "product" in self.request.match_info or version in ("latest", "stable"):
            raise web.HTTPMethodNotAllowed("PUT", ["GET"])
        fmt = _get_format(self.repo(), self.request.match_info)
        try:
            internal = (await self.json_body())["artifact_format"]["internal"]
            if not isinstance(internal, bool):
                raise TypeError()
        except (TypeError, KeyError) as e:
            raise web.HTTPBadRequest() from e
        fmt.set_internal(internal)
        return web.Response()

    async def patch(self):
        """
        Remove the dirty flag from a format.
diff --git a/dlrepo/views/job.py b/dlrepo/views/job.py
index 49fbbb3bc97f..79e5dddbe6ca 100644
--- a/dlrepo/views/job.py
+++ b/dlrepo/views/job.py
@@ -81,7 +81,6 @@ class JobView(JobArchiveView):
                    formats.append(
                        {
                            "name": f.name,
                            "internal": f.is_internal(),
                            "rpm": rpm,
                            "deb": deb,
                            "url": fmt_url,
diff --git a/dlrepo/views/product.py b/dlrepo/views/product.py
index 2d8381c3208b..9a0a11289bd2 100644
--- a/dlrepo/views/product.py
+++ b/dlrepo/views/product.py
@@ -203,7 +203,6 @@ class VersionView(BaseView):
                    formats.append(
                        {
                            "name": fmt.name,
                            "internal": fmt.is_internal(),
                            "rpm": rpm,
                            "deb": deb,
                            "url": fmt_url,
diff --git a/dlrepo/views/util.py b/dlrepo/views/util.py
index d6c92bc85cdf..b5404cc89825 100644
--- a/dlrepo/views/util.py
+++ b/dlrepo/views/util.py
@@ -67,7 +67,6 @@ class BaseView(web.View):
            "artifact_format": {
                "name": fmt.name,
                "relpath": relpath.rstrip("/"),
                "internal": fmt.is_internal(),
                "dirs": dirs,
                "files": files,
            },
diff --git a/docs/dlrepo-api.7.scdoc b/docs/dlrepo-api.7.scdoc
index 79db971036e3..0173ef9f0fdc 100644
--- a/docs/dlrepo-api.7.scdoc
+++ b/docs/dlrepo-api.7.scdoc
@@ -595,7 +595,6 @@ Get the list of files present in the specified _{format}_.
	{
		"artifact_format": {
			"name": "docs",
			"internal": false,
			"dirty": false,
			"files": [
				"config/index.html",
@@ -672,30 +671,6 @@ Check if the specified _{format}_ exists and is not _dirty_.
	- _404_: the specified _{format}_ does not exist or has the _dirty_ flag
	  set to _true_.

## PUT /branches/{branch}/{tag}/{job}/{format}/
## PUT /products/{product}/{variant}/{product_branch}/{version}/{format}/
## PUT /~{user}/branches/{branch}/{tag}/{job}/{format}/
## PUT /~{user}/products/{product}/{variant}/{product_branch}/{version}/{format}/

Set the _internal_ flag for the specified _{format}_. This flag is only
informative and does not imply any restrictions about the accessibility of the
format. Use *dlrepo-acls*(5) to restrict access to artifact formats.

*Access:*
	_rw_
*Request:*
	```
	{
		"artifact_format": {
			"internal": true
		}
	}
	```
*Errors:*
	- _400_: invalid parameters.
	- _404_: the specified _{tag}_ or _{version}_ does not exist.
	- _405_: _{tag}_ or _{version}_ is either _latest_ or _stable_.

## PATCH /branches/{branch}/{tag}/{job}/{format}/
## PATCH /products/{product}/{variant}/{product_branch}/{version}/{format}/
## PATCH /~{user}/branches/{branch}/{tag}/{job}/{format}/
diff --git a/docs/dlrepo-cli.1.scdoc b/docs/dlrepo-cli.1.scdoc
index dd383a1eba21..7df67b8a5b1f 100644
--- a/docs/dlrepo-cli.1.scdoc
+++ b/docs/dlrepo-cli.1.scdoc
@@ -186,23 +186,6 @@ _BRANCH_
_TAG_
	The tag name.

## dlrepo-cli internal [-u] BRANCH TAG JOB FORMAT

Set or unset the _internal_ status on an artifact format.

*-u*, *--unset*
	Unset the _internal_ status from the format instead of setting it.

_BRANCH_
	The branch name.
_TAG_
	The tag name.
_JOB_
	The job name.
_FORMAT_
	The artifact format name.


## dlrepo-cli delete [-f] BRANCH [TAG] [JOB]

Delete a job, a tag or a branch and all its tags recursively.
diff --git a/docs/dlrepo-layout.7.scdoc b/docs/dlrepo-layout.7.scdoc
index 2e8d993588a3..c26e58a1939c 100644
--- a/docs/dlrepo-layout.7.scdoc
+++ b/docs/dlrepo-layout.7.scdoc
@@ -149,9 +149,6 @@ _.dirty_
	file is exists, _HEAD_ requests to this artifact format will return
	a _404 Not found_ error (see *dlrepo-api*(7)). This file is deleted
	after a _PATCH_ request is made on this format.
_.internal_ (optional)
	Empty file created when the artifact format is marked as "internal".
	This flag is only informative.

## products/

diff --git a/scss/sections.scss b/scss/sections.scss
index 0c7ec16ffcd5..02002bd410c3 100644
--- a/scss/sections.scss
+++ b/scss/sections.scss
@@ -104,12 +104,6 @@ section.files {
    > .artifact-format:hover {
      background-color: lighten($lightgray, 5%);
    }
    > .artifact-format.internal {
      border-color: #fca5a5;
    }
    > .artifact-format.internal:hover {
      background-color: #fee2e2;
    }
  }

  > .whole-archive {
-- 
2.30.2

[PATCH dlrepo v3 6/7] tag: clean published jobs before re-publishing

Details
Message ID
<20220516101638.10977-7-julien.floret@6wind.com>
In-Reply-To
<20220516101638.10977-1-julien.floret@6wind.com> (view parent)
DKIM signature
missing
Download raw message
Patch: +3 -1
Sometimes, after a tag has been published to a remote server, we need to
rebuild the local tag and publish it again, because something was wrong
in the original build. The caveat is that when re-publishing a job, any
file in the original job that does not exist anymore in the updated job
will not be removed from the remote server. This is especially
problematic when re-publishing a container: since the manifest file is
named after its sha256 sum, we end up with two manifest files on the
remote server and docker access is denied.

Force deletion of jobs already published before re-publishing them.

Signed-off-by: Julien Floret <julien.floret@6wind.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>
---
 dlrepo/fs/tag.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/dlrepo/fs/tag.py b/dlrepo/fs/tag.py
index 33def160681e..40a0bfcf6817 100644
--- a/dlrepo/fs/tag.py
+++ b/dlrepo/fs/tag.py
@@ -134,6 +134,9 @@ class Tag(SubDir):
        for job in self.get_jobs():
            if job.is_internal():
                continue
            job_url = job.url()
            # in case job is already present on the server, delete it before re-uploading it
            await sess.delete(job_url, params={"force": "true"}, raise_for_status=False)
            self._publish_status_path().write_text(f"uploading {job.name}\n")
            tasks = []
            for fmt in job.get_formats():
@@ -142,7 +145,6 @@ class Tag(SubDir):
            metadata = job.get_metadata()
            del metadata["name"]
            del metadata["locked"]
            job_url = job.url()
            LOG.debug("publishing job metadata %s", job_url)
            async with semaphore:
                await sess.patch(job_url, json={"job": metadata})
-- 
2.30.2

[PATCH dlrepo v3 5/7] job: display upload time

Details
Message ID
<20220516101638.10977-6-julien.floret@6wind.com>
In-Reply-To
<20220516101638.10977-1-julien.floret@6wind.com> (view parent)
DKIM signature
missing
Download raw message
Patch: +109 -2
Add the job upload time (as an integer timestamp) to the job metadata.
Its value is determined by looking at the modification time of a
.stamp file, which is created when the job is uploaded.

The upload time is then be displayed on the job web page, in
a user-friendly format indicating the user's local time.

Signed-off-by: Julien Floret <julien.floret@6wind.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>
---
 dlrepo/fs/container.py     |  1 +
 dlrepo/fs/fmt.py           |  2 +-
 dlrepo/fs/job.py           | 21 ++++++++++++
 dlrepo/templates/job.html  |  5 +++
 dlrepo/views/__init__.py   | 11 ++++++-
 dlrepo/views/job.py        |  1 +
 dlrepo/views/localtime.py  | 66 ++++++++++++++++++++++++++++++++++++++
 docs/dlrepo-api.7.scdoc    |  1 +
 docs/dlrepo-layout.7.scdoc |  3 ++
 9 files changed, 109 insertions(+), 2 deletions(-)
 create mode 100644 dlrepo/views/localtime.py

diff --git a/dlrepo/fs/container.py b/dlrepo/fs/container.py
index 86c38baffd08..4a6dde01e6d9 100644
--- a/dlrepo/fs/container.py
+++ b/dlrepo/fs/container.py
@@ -99,6 +99,7 @@ class ContainerRegistry:
        # manifest, we can read their size on the filesystem.

        # link config blob into the job folder
        job.create()
        config = manifest.get("config", {})
        config["size"] = self._link_blob_to_job(config.get("digest"), job)

diff --git a/dlrepo/fs/fmt.py b/dlrepo/fs/fmt.py
index 29c372e0627e..6d67ffbeb331 100644
--- a/dlrepo/fs/fmt.py
+++ b/dlrepo/fs/fmt.py
@@ -133,7 +133,7 @@ class ArtifactFormat(SubDir):
    def set_dirty(self, dirty: bool):
        path = self._dirty_path()
        if dirty:
            path.parent.mkdir(mode=0o755, parents=True, exist_ok=True)
            self.create()
            path.touch()
        elif path.is_file():
            path.unlink()
diff --git a/dlrepo/fs/job.py b/dlrepo/fs/job.py
index eb37e32c276f..5120f159f5f9 100644
--- a/dlrepo/fs/job.py
+++ b/dlrepo/fs/job.py
@@ -23,6 +23,26 @@ class Job(SubDir):
    TODO
    """

    def create(self):
        super().create()
        stamp = self._path / ".stamp"
        if not stamp.exists():
            stamp.touch()

    @classmethod
    def creation_date(cls, j):
        stamp = j.path() / ".stamp"
        if stamp.is_file():
            # prefer mtime over ctime
            # on UNIX, ctime is "the time of most recent metadata change" whereas
            # mtime is "most recent content modification"
            return stamp.stat().st_mtime
        return 0

    @property
    def timestamp(self) -> int:
        return Job.creation_date(self)

    def get_formats(self) -> Iterator[ArtifactFormat]:
        yield from ArtifactFormat.all(self)

@@ -121,6 +141,7 @@ class Job(SubDir):
    def set_locked(self, locked: bool):
        if not self.exists():
            raise FileNotFoundError()
        self.create()
        path = self._lock_path()
        if locked:
            path.touch()
diff --git a/dlrepo/templates/job.html b/dlrepo/templates/job.html
index e5ecc6719456..bd6ca42bb17d 100644
--- a/dlrepo/templates/job.html
+++ b/dlrepo/templates/job.html
@@ -6,6 +6,7 @@

{% set job_name = job.pop("name") %}
{% set formats = job.pop("artifact_formats", []) %}
{% set timestamp = job.pop("timestamp") %}

{% block breadcrumbs %}
{{breadcrumbs(['branches', branch, tag, job_name])}}
@@ -47,6 +48,10 @@
    <span class="value">{{value}}</span>
  </div>
  {% endfor %}
  <div class="metadata">
    <span class="key">upload_time</span>
    <span class="value">{{timestamp|local_time}}</span>
  </div>
</section>

{% if job.product and job.product_variant and job.product_branch and job.version %}
diff --git a/dlrepo/views/__init__.py b/dlrepo/views/__init__.py
index 85b5b7b44321..1eb76975117c 100644
--- a/dlrepo/views/__init__.py
+++ b/dlrepo/views/__init__.py
@@ -3,7 +3,7 @@
# SPDX-License-Identifier: BSD-3-Clause

import asyncio
from datetime import datetime
from datetime import datetime, timezone
import os
import pathlib

@@ -11,6 +11,7 @@ from aiohttp import web
import aiohttp_jinja2
import jinja2

from . import localtime
from ..fs.util import human_readable
from .artifact import ArtifactView
from .branch import BranchesView, BranchView
@@ -170,6 +171,13 @@ async def template_vars(request):

# --------------------------------------------------------------------------------------
def add_routes(app):
    def pretty_localtime(timestamp, fmt="%Y-%m-%d %H:%M:%S UTC%z"):
        if not timestamp:
            return "n/a"
        utc_time = datetime.fromtimestamp(timestamp, timezone.utc)
        local_time = utc_time.astimezone(localtime.LOCAL)
        return local_time.strftime(fmt)

    template_dirs = []
    if os.getenv("DLREPO_TEMPLATES_DIR"):
        template_dirs.append(os.getenv("DLREPO_TEMPLATES_DIR"))
@@ -188,6 +196,7 @@ def add_routes(app):
        extensions=["jinja2.ext.do"],
        trim_blocks=True,
        lstrip_blocks=True,
        filters={"local_time": pretty_localtime},
    )
    for route in (
        HomeView,
diff --git a/dlrepo/views/job.py b/dlrepo/views/job.py
index 5297754ae6ee..9ad32e43b462 100644
--- a/dlrepo/views/job.py
+++ b/dlrepo/views/job.py
@@ -68,6 +68,7 @@ class JobView(JobArchiveView):
        html = "html" in self.request.headers.get("Accept", "json")
        data = {"job": job.get_metadata()}
        data["job"]["internal"] = job.is_internal()
        data["job"]["timestamp"] = job.timestamp
        formats = []
        for f in job.get_formats():
            fmt_url = f.url()
diff --git a/dlrepo/views/localtime.py b/dlrepo/views/localtime.py
new file mode 100644
index 000000000000..f1ad770d6521
--- /dev/null
+++ b/dlrepo/views/localtime.py
@@ -0,0 +1,66 @@
# Code imported from python official documentation
# https://docs.python.org/3.7/library/datetime.html

from datetime import datetime, timedelta, tzinfo
import time as _time


ZERO = timedelta(0)
HOUR = timedelta(hours=1)
SECOND = timedelta(seconds=1)

# A class capturing the platform's idea of local time.
# (May result in wrong values on historical times in
#  timezones where UTC offset and/or the DST rules had
#  changed in the past.)

STDOFFSET = timedelta(seconds=-_time.timezone)
if _time.daylight:
    DSTOFFSET = timedelta(seconds=-_time.altzone)
else:
    DSTOFFSET = STDOFFSET

DSTDIFF = DSTOFFSET - STDOFFSET


class LocalTimezone(tzinfo):
    def fromutc(self, dt):
        assert dt.tzinfo is self
        stamp = (dt - datetime(1970, 1, 1, tzinfo=self)) // SECOND
        args = _time.localtime(stamp)[:6]
        dst_diff = DSTDIFF // SECOND
        # Detect fold
        fold = args == _time.localtime(stamp - dst_diff)
        return datetime(*args, microsecond=dt.microsecond, tzinfo=self, fold=fold)

    def utcoffset(self, dt):
        if self._isdst(dt):
            return DSTOFFSET
        return STDOFFSET

    def dst(self, dt):
        if self._isdst(dt):
            return DSTDIFF
        return ZERO

    def tzname(self, dt):
        return _time.tzname[self._isdst(dt)]

    def _isdst(self, dt):
        tt = (
            dt.year,
            dt.month,
            dt.day,
            dt.hour,
            dt.minute,
            dt.second,
            dt.weekday(),
            0,
            0,
        )
        stamp = _time.mktime(tt)
        tt = _time.localtime(stamp)
        return tt.tm_isdst > 0


LOCAL = LocalTimezone()
diff --git a/docs/dlrepo-api.7.scdoc b/docs/dlrepo-api.7.scdoc
index 018d6833822a..4e75f883e61b 100644
--- a/docs/dlrepo-api.7.scdoc
+++ b/docs/dlrepo-api.7.scdoc
@@ -313,6 +313,7 @@ Get metadata and artifact formats for the specified _{job}_.
			"name": "foobaz-x86_64-postgresql",
			"locked": true,
			"internal": false,
			"timestamp": 1637001613.5863628,
			"product": "foobaz",
			"product_variant": "x86_64-postgresql",
			"product_branch": "3.2",
diff --git a/docs/dlrepo-layout.7.scdoc b/docs/dlrepo-layout.7.scdoc
index 7fcd2ef49b1f..49ab60062105 100644
--- a/docs/dlrepo-layout.7.scdoc
+++ b/docs/dlrepo-layout.7.scdoc
@@ -112,6 +112,9 @@ _.metadata_ (optional)
		"baz": "bar"
	}
	```
_.stamp_
	Empty file created when the job is first created. This is used to
	display the job creation time.
_.locked_ (optional)
	Empty file created when the job is locked to avoid further
	modifications. A locked job cannot receive file uploads nor metadata
-- 
2.30.2

[PATCH dlrepo v3 7/7] tag: robustify TagView.put()

Details
Message ID
<20220516101638.10977-8-julien.floret@6wind.com>
In-Reply-To
<20220516101638.10977-1-julien.floret@6wind.com> (view parent)
DKIM signature
missing
Download raw message
Patch: +16 -5
Check the types of the input parameters: when at least one has not the
right type, respond with HTTPBadRequest and apply none (previously,
parameters of the wrong type were silently ignored.)
Also catch the FileNotFoundError exception to respond a proper
HTTPNotFound.

Signed-off-by: Julien Floret <julien.floret@6wind.com>
Acked-by: Olivier Matz <olivier.matz@6wind.com>
---
 dlrepo/views/tag.py | 21 ++++++++++++++++-----
 1 file changed, 16 insertions(+), 5 deletions(-)

diff --git a/dlrepo/views/tag.py b/dlrepo/views/tag.py
index 7430269f1e2e..2d20bfe6f9a8 100644
--- a/dlrepo/views/tag.py
+++ b/dlrepo/views/tag.py
@@ -73,13 +73,24 @@ class TagView(BaseView):
        tag = self._get_tag()
        try:
            data = (await self.json_body())["tag"]
            released = data.get("released")
            if released is not None and not isinstance(released, bool):
                raise TypeError()
            locked = data.get("locked")
            if locked is not None and not isinstance(locked, bool):
                raise TypeError()
        except (TypeError, KeyError) as e:
            raise web.HTTPBadRequest(reason="invalid parameters") from e
        if isinstance(data.get("released"), bool):
            semaphore = self.request.app["dlrepo_publish_semaphore"]
            tag.set_released(data["released"], semaphore)
        if isinstance(data.get("locked"), bool):
            tag.set_locked(data["locked"])

        try:
            if released is not None:
                semaphore = self.request.app["dlrepo_publish_semaphore"]
                tag.set_released(released, semaphore)
            if locked is not None:
                tag.set_locked(locked)
        except FileNotFoundError as e:
            raise web.HTTPNotFound() from e

        return web.Response()

    async def delete(self):
-- 
2.30.2
Details
Message ID
<CK3QIOG6VOD8.3Q58CTXJ6OF44@marty>
In-Reply-To
<20220516101638.10977-1-julien.floret@6wind.com> (view parent)
DKIM signature
missing
Download raw message
Julien Floret, May 16, 2022 at 12:16:
> This series is motivated 
> This v3 is sent because the git server did not recognized the v2 because
> of an unfortunate in-reply-to. Otherwise, no change since v2. 
>
> - the first patch does some cleanup in dlrepo-cli.
> - patches 2 to 4 add an "internal" flag to the job to prevent jobs meant
>   for internal testing to be published to an external server. The
>   "internal" flag on formats was confusing, it is dropped.
> - patch 5 exports the job upload timestamp in the metadata, to display
>   it on the html page.
> - patch 6 ensures that when re-publishing a job it does not keep stale
>   files from the previous publishing
> - patch 7 adds some checks to TagView.put(), inspired by changes done in
>   JobView.put() in patch 3.
>
> Julien Floret (7):
>   cli: remove unused local_dir() function
>   fmt: remove internal flag
>   job: add internal flag
>   tests: check internal jobs are not published
>   job: display upload time
>   tag: clean published jobs before re-publishing
>   tag: robustify TagView.put()
>
>  dlrepo-cli                                    | 17 ++---
>  dlrepo/fs/container.py                        |  1 +
>  dlrepo/fs/fmt.py                              | 19 +-----
>  dlrepo/fs/job.py                              | 35 ++++++++++
>  dlrepo/fs/tag.py                              | 11 ++--
>  dlrepo/templates/job.html                     | 10 +--
>  dlrepo/templates/product_version.html         |  5 +-
>  dlrepo/views/__init__.py                      | 11 +++-
>  dlrepo/views/fmt.py                           | 20 ------
>  dlrepo/views/job.py                           | 19 ++++--
>  dlrepo/views/localtime.py                     | 66 +++++++++++++++++++
>  dlrepo/views/product.py                       |  1 -
>  dlrepo/views/tag.py                           | 21 ++++--
>  dlrepo/views/util.py                          |  1 -
>  docs/dlrepo-api.7.scdoc                       | 33 ++--------
>  docs/dlrepo-cli.1.scdoc                       | 10 ++-
>  docs/dlrepo-layout.7.scdoc                    |  8 ++-
>  scss/sections.scss                            |  6 --
>  .../branch/tag/internal_job/.internal         |  0
>  .../branch/tag/internal_job/fmt1/.digests     |  1 +
>  .../branches/branch/tag/internal_job/fmt1/a   |  1 +
>  .../branches/branch/tag/internal_job/fmt1/b   |  1 +
>  tests/test_publish.py                         |  5 ++
>  23 files changed, 185 insertions(+), 117 deletions(-)
>  create mode 100644 dlrepo/views/localtime.py
>  create mode 100644 tests/publish_reference/branches/branch/tag/internal_job/.internal
>  create mode 100644 tests/publish_reference/branches/branch/tag/internal_job/fmt1/.digests
>  create mode 100644 tests/publish_reference/branches/branch/tag/internal_job/fmt1/a
>  create mode 100644 tests/publish_reference/branches/branch/tag/internal_job/fmt1/b

Hey there,

I only had one remark about the job upload date formatting. In my
opinion, it is not worth it to display it using the server's timezone.
It adds complexity, code and will only be meaningful to users that are
in the same timezone. I pushed the whole series with that patch edited
to display the time in UTC.

Thanks a lot!

Acked-by: Robin Jarry <robin@jarry.cc>
Reply to thread Export thread (mbox)