[WIP] Apply patches from Patchwork payload

[yurinnick]

Fixes kernelci/kernelci-api#307
Depends on kernelci/kernelci-core#2041

Looks up for Patchwork payload in node.data and apply patches by mbox url before packaging tree into a tarball.
We expect following minimal structure for Patchwork payload:

{
    "version": 1,
    "patches": [
      {
        "id": int,
        "hash": string,
        "web_url": string,
        "date": datetime,
        "mbox": string
      }
    ]
}

Test run logs:

kernelci-pipeline-notifier            | 07/31/2023 11:33:19 PM UTC [INFO] 2023-07-31 23:33:19.800845  HEAD          64c844bfe8ad32503e18af61  Running    -         checkout
kernelci-pipeline-tarball             | 07/31/2023 11:33:19 PM UTC [INFO] Updating repo for mainline
kernelci-pipeline-tarball             | From https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux
kernelci-pipeline-tarball             |  * branch                      HEAD       -> FETCH_HEAD
kernelci-pipeline-tarball             | From git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux
kernelci-pipeline-tarball             |  * branch                      HEAD       -> FETCH_HEAD
kernelci-pipeline-tarball             | HEAD is now at 5d0c230f1de8 Linux 6.5-rc4
kernelci-pipeline-tarball             | 07/31/2023 11:33:27 PM UTC [INFO] Repo updated
kernelci-pipeline-tarball             | 07/31/2023 11:33:27 PM UTC [INFO] Applying patch to the repo, patch mbox url: https://patchwork.kernel.org/project/linux-hardening/patch/20230726231139.never.601-kees@kernel
.org/mbox/
kernelci-pipeline-tarball             | 07/31/2023 11:33:28 PM UTC [INFO] Making tarball linux-mainline-master-v6.5-rc4-1-g66f6b7deab6b.tar.gz
kernelci-pipeline-tarball             | 07/31/2023 11:33:28 PM UTC [INFO] set -e
kernelci-pipeline-tarball             | cd /home/kernelci/data/src/linux
kernelci-pipeline-tarball             | git archive --format=tar --prefix=linux-mainline-master-v6.5-rc4-1-g66f6b7deab6b/ HEAD | gzip > ../../output/linux-mainline-master-v6.5-rc4-1-g66f6b7deab6b.tar.gz
kernelci-pipeline-tarball             |
kernelci-pipeline-tarball             |
kernelci-pipeline-tarball             | 07/31/2023 11:33:55 PM UTC [INFO] Tarball created

[gctucker]

Where is the Patchwork data coming from? In other words, would be sending a Node object with that data to the API?

[gctucker]

One issue there is the git describe, as it will be different every time the patches get applied. It's meant to be the Git revision from the upstream repository, in this case it won't refer to anything usable with the patches added as local commits.

This is the part we need to clarify I think. Initially, we could just keep the git describe and sha1 values as the base revision from the repository. Then a child node could be added with meta-data about the patchset and a new tarball with the patches applied for example. There are different ways to design this, it's important because of the .revision field which is part of the Node schema. So we can't keep everything in .data as reporting will be confusing with mismatching git revisions.

I'll create a GitHub issue to cover this part, then once we have at least one functional implementation we can revisit this PR and update it accordingly. Looking at how KCIDB is handling patchsets should also help there, like we discussed in the meeting last Wednesday.

[yurinnick]

So should we extend .revision to include patchset hash? What should be patchset hash?
I think my proposal would be implement patch/patchset hashing based on patchwork hasher and CKI approach described here:

• For patch hash will be sha256 (sha1?) hash of normalized patch content
• for patchset hash will be sha256(sha1?) hash of all patches in order they were applied

This way we'll store unique pair of base commit revision and patchset hash in the Node.revision.

[gctucker]

Yes I think the patches have to be part of the revision data and not just some extra arbitrary data. It would ideally need to be compatible with KCIDB, either by using the exact same schema and mechanism or by having a clear translation mechanism.

There are different ways to implement this, and I think it can be done incrementally. I'll post some comments on kernelci/kernelci-api#307

[yurinnick]

Replace it with Patchset service implementation: #342