[RFC] Patch building and Webhooks API

[yurinnick]

Webhook API should help to integrate KernelCI with patch management (Patchwork) and version control systems (Github, Gitlab). It will provide interface to trigger non-upstream patch builds that should be able to publish results back later.

Implementation

As the first step, I'd like to implement Patchwork integration with KernelCI. Following changes should be implemented:

1. Add /webhooks/patchwork API that will expect certain (TBD) input from Patchwork side, enough to build and test kernel, and report back results
2. Extend kernelci.Node, KernelBuildMetadata, kernelci.config.Tree with patch-related fields
3. Implement patch mbox application on top of git checkout
4. Implement Patchwork patch checks update mechanism

Minimal patch information

We need a patch and all dependent patches information, as well as submitter information for email notifications.

{
  "patches": [
    {
      "patchwork_id": "str",
      "hash": "str",
      "web_url": "str",
      "date": "str",
      "mbox": "str"
    }
  ],
  "submitter": { 
    "id": "int", 
    "url": "str", 
    "name": "str", 
    "email": "str" 
  }
}

Discussion

• How to make patches information generic enough to share across multiple systems? Should we make it generic at all?
• How to pass Patchwork specific metadata into Node? data field or a separate field?
• Calculating and passing revision information
• What data should we expect in /webhooks/patchwork? (Collaboration with Patchwork developers)

[gctucker]

Add /webhooks/patchwork API that will expect certain (TBD) input from Patchwork side, enough to build and test kernel, and report back results

Do you mean adding an endpoint to the KernelCI API?

I think a better approach is to have a micro-service receiving the webhooks from Patchwork and then relaying things to the KernelCI API. We have something like this already for LAVA callbacks:
https://github.com/kernelci/kernelci-pipeline/blob/main/src/lava_callback.py

That way, the KernelCI API doesn't need to "know" about Patchwork and we can update the bridge any time just by restarting the client-side service. It's a key principle of the new API, to be modular and leave all the specific logic happen on the client side.

[gctucker]

How to pass Patchwork specific metadata into Node? data field or a separate field?

Yes, the data field is like a void * data structure managed by the client-side logic. The database is NoSQL (MongoDB) so there's no schema for the payload stored in data. All the other fields are validated through a schema though. A bit like a file system, the meta-data is structured by the kernel but what goes into the files is managed by users - if that's a suitable enough analogy.

[gctucker]

How to make patches information generic enough to share across multiple systems? Should we make it generic at all?

I would suggest to make it entirely specific to Patchwork and whatever data BPF CI currently has, with a dedicated micro-service just for this use-case. This would be the first time we add "pre-merge" testing capability to KernelCI so we can keep things as simple as possible and tailored to this particular use-case. Then when we start adding other kinds of pre-merge testing sources we might have other dedicated micro-services for them and potentially consolidate things with a more standard way of triggering pipeline based on patch series, but I don't think we need to solve this kind of generic problem right now just for BPF CI integration.

[yurinnick]

Going dedicated microservice route will certainly help with iterations and decoupling logic from the core part. But now I am thinking about how to handle sending information back to patchworks?
We can store mapping patch_id<>pipeline_id on "patchwork webhook microservice" side and poll results from there. Or we can pass patch metadata into free-form data field and create special patchwork notifier on kernelci-pipeline side.
Latter one is easier, but it make kernelci-pipeline at least somewhat patchwork-aware.

[yurinnick]

As for BPF CI, we already have CI setup and ready to send results into KCIDB. There are a few missing pieces there, mainly missing patch support in KCIDB and WebUI.

[gctucker]

Going dedicated microservice route will certainly help with iterations and decoupling logic from the core part.

Yes, it's one of the key design principles of the new API. The legacy API had too many custom built-in features and that made it very hard to do anything with it. Testing the upstream kernel requires a very modular design as each use-case is different - here it's Patchwork for BPF but nearly every maintainer has some custom tooling.

But now I am thinking about how to handle sending information back to patchworks? We can store mapping patch_id<>pipeline_id on "patchwork webhook microservice" side and poll results from there. Or we can pass patch metadata into free-form data field

Yes, the latter. It's much better to keep the micro-service stateless, the Node.data field can be used for this kind of extra data.

and create special patchwork notifier on kernelci-pipeline side. Latter one is easier, but it make kernelci-pipeline at least somewhat patchwork-aware.

This is fine, in fact the micro-service doesn't even need to be in the kernelci-pipeline repository. It could be something hosted as part of BPF CI, or it could be in kernelci-pipeline with a separate docker-compose file like we have for the LAVA callback handler. Having a micro-service dedicated to BPF patch testing is absolutely the kind of thing the new API was designed for.

[gctucker]

As for BPF CI, we already have CI setup and ready to send results into KCIDB. There are a few missing pieces there, mainly missing patch support in KCIDB and WebUI.

OK, I guess it'll be good to clarify what you want to cover with your existing CI and what you're expecting from KernelCI to be doing. If you want the KernelCI infrastructure to build kernels and run tests for BPF patches, then would it be duplicating some things you're already doing with your current BPF CI? Would it be a way to transition from BPF CI to KernelCI?

[yurinnick]

In kcidb/orm/data.py there patchset_hash and patchset_files fields for checkout and revision available in KCIDB. I can't find any mentions of revision structure in the docs though, so I assume it's not used.

Does KernelCI store any patchset information like mbox urls and/or title? If not, what's the best place to add these fields? KCIDB itself?

[yurinnick]

If you want the KernelCI infrastructure to build kernels and run tests for BPF patches, then would it be duplicating some things you're already doing with your current BPF CI?

Part of the reason I want this functionality is to reduce BPF CI footprint. I'd like to rely on KernelCI as much as possible in the future, while keeping more Meta-workload specific BPF tests on BPF CI side.

Would it be a way to transition from BPF CI to KernelCI?

Once we'll have patch/patchset abstraction in KCIDB and Dashboards we'll publish data from BPF CI into KernelCI. Then we can start removing duplicated signals and move some BPF CI testing onto KernelCI infrastructure.

[gctucker]

@yurinnick I think we need to clarify a few things, KCIDB is only a database for storing data from any CI system. It doesn't run builds or tests or anything itself. The point of it is to have a common email report (and dashboard) with results aggregated from many CI systems e.g. 0-Day, Syzbot, CKI, Gentoo, Linaro's TuxSuite as well as the "native" KernelCI results.

The "native" KernelCI results are the builds, tests and anything that is orchestrated by KernelCI itself. There's a legacy database and frontend for it and it's running with Jenkins. This is now being replaced with the new API & Pipeline which will have a new web dashboard too at some point.

So you could already choose to send your current BPF CI results to KCIDB. I'm not an expert regarding the schema and how it handles patch sets, that's more a question for @spbnick.

Then if you want to rely on KernelCI to run tests for BPF, that's not related to KCIDB. It's something we can do with the new API & Pipeline by setting up a dedicated pipeline service to make the bridge between the patch sets you have to test for BPF and KernelCI. I believe this is what you wanted to do, and offload your current CI system or maybe refocus it on some "internal" testing while KernelCI would be gradually covering all the upstream-oriented testing. Is that right?

I'm not familiar enough yet with your setup around Patchwork and GitHub but we can go through the details if integrating it with KernelCI is what you want to achieve.

[yurinnick]

As part of proof-of-concept, added patchwork functionality to various kernelci parts:

• src/send_kcidb.py: Extract patchset hash from patchwork metadata kernelci-pipeline#299
• [WIP] src/test_report.py: Include patchwork patch metadata into test report kernelci-pipeline#297
• [WIP] Apply patches from Patchwork payload kernelci-pipeline#295
• kernelci/build.py: Add apply_patch_mbox method kernelci-core#2041

[gctucker]

Could you please explain which service is going to be receiving what kind of data from Patchwork to start with?

We basically need something to know the Git base revision and a list of patch files. Then the simplest way to deal with this would be to apply the patches on top of the git revision and create a source tarball "checkout" with it, and the rest can remain unchanged. Once we have this in place, I think we can gradually add more features specific to patches like keeping them as separate files, doing incremental builds etc. but that will require many development iterations I think.