Generalized collections

[chaoran-chen]

On CoV-Spectrum, a popular feature are the collections. Any user can create a collection which mainly consists of

• Title
• Description
• List of variant entry where each only consists of
  • Name
  • A normal variant query or an advanced variant query (the SARS-CoV-2-specific query, not the new advanced query)
  • Description

We want to have this feature for all of GenSpectrum, for all organisms.

Data structure / API

Like with CoV-Spectrum collections, we want to support multiple ways of specifying a variant. This can either be a lineage filter + a set of mutations, or an entirely custom advanced query. For the advanced queries, not all filters are valid, for example, you shouldn't be able to filter on collection dates.

In the database, we want to store these different objects as a tagged union.

Authentication

Unlike with CoV-Spectrum, for GS we want to use user account authentication, starting with GitHub as our identity provider.

We want to still implement a mechanism where users can easily collaborate on collections, either by having a collection-admin create edit tokens, or inviting other users as collaborators (Similar to how in Google Docs you can invite someone specifically, or share a general edit link).

We also want to offer a way to authenticate/provide a token that's easy to do automated. We know that users use scripts to update collections, and we want to continue to support this.

We also considered login with ORCID: #1046

User interface

To be refined

Steps

To generalize the feature for whole GenSpectrum / different pathogens, we need:

• Implement an API to create/edit collections in the backend
  • Implement initial generalized collections in the backend #1045
  • The backend is proxied through the frontend server, so we can hide the new endpoints for now, if we wanted to (until we have auth implemented).
• Define an authentication mechanism (Do we do it the same way as with the CoV-Spectrum collections?)
• Create a web interface to create and maintain collections

[fhennig]

Hey! Good to have this issue, also sounds like a useful feature!

I think you can also just have a query page with the URL encoded query, and you could share that as well (obv. with some size restrictions for the query). But this is persistent and named, and thus more easily shareable, which is nice to have, right?

To create a CoV-Spectrum collection, a user does not need to register but get an admin URL that consists of a key.

To clarify: You can just create a collection, and when you get it you get an admin URL (some UUID or similar) that you can use to edit the collection in the future. If you lose the URL, you can't edit it anymore.

Cool, because no login, no user management, great. And I suppose you didn't have many issues with people loosing the URLs or something, right?

Implement an API to create/edit collections in the backend

The README shows that the backend is currently not in use, we'd first have to reactive it, right?

[chaoran-chen]

I think you can also just have a query page with the URL encoded query, and you could share that as well (obv. with some size restrictions for the query). But this is persistent and named, and thus more easily shareable, which is nice to have, right?

That might also be useful, but for CoV-Spectrum, people like to have stable URLs/pages that are regularly updated and used by many people. For example, you can find many mentions of "collection42" (see Twitter) that refer to the collection with ID 42 on CoV-Spectrum. For #939, we'd also need persist the collections.

To clarify: You can just create a collection, and when you get it you get an admin URL (some UUID or similar) that you can use to edit the collection in the future. If you lose the URL, you can't edit it anymore.

Cool, because no login, no user management, great. And I suppose you didn't have many issues with people loosing the URLs or something, right?

ok, what I left out is that you can also specify an author name and email address. We don't have any automated reset mechanism but there were cases where people contacted me via email and I sent them the admin link.

The README shows that the backend is currently not in use, we'd first have to reactive it, right?

Yes!

[fhennig]

Use case: As a user, I want to be able to create a collection with all the mutations that create resistance to antibodies. They are defined here: https://viralzone.expasy.org/11605 (or as a TSV here: https://viralzone.expasy.org/resources/RSV/F%5FRSV%5Fhuman.txt)

I want the definition of the collection to be relatively simple -- maybe without having to write a query? And I'm not sure if I want it to be easily editable afterwards, maybe that's not required.

For the implementation, maybe that'd mean we need a good "wizard" to create collections, or we need to be able to store the collection as a list of mutations and then generate the query when we need to. Not sure if this might be overcomplicating things.

[fhennig]

Nuclino page for discussion on this issue: https://app.nuclino.com/GenSpectrum/Development/Generalized-collections-e11e1763-e7b4-43ff-b407-db8064356155