feat: add RSS data models (SQLAlchemy + Pydantic) by HeloiseJoffe · Pull Request #857 · DIRACGrid/diracx

HeloiseJoffe · 2026-03-25T09:03:58Z

Closes #837

Changes :

Add RSS SQLAlchemy 2.0 models for the tables and ResourceStatusDB class
Add RSS entry point
Add pydantic models and ResourceType enum
Add Unit tests

diracx-core/src/diracx/core/models/rss.py

diracx-db/src/diracx/db/sql/rss/schema.py

diracx-core/src/diracx/core/models/rss.py

read-the-docs-community · 2026-03-26T16:30:37Z

Documentation build overview

📚 diracx | 🛠️ Build #32165038 | 📁 Comparing 5c3f59d against latest (63dc086)

🔍 Preview build

No files changed.

diracx-db/src/diracx/db/sql/rss/__init__.py

diracx-db/tests/rss/test_rss_db.py

diracx-db/src/diracx/db/sql/rss/schema.py

diracx-db/src/diracx/db/sql/rss/db.py

aldbr · 2026-03-31T08:56:52Z

diracx-db/src/diracx/db/sql/rss/db.py

Right now db.py does three things:

queries the database

dispatches based on resource type

converts DB strings into Pydantic models via map_status().

In diracx's architecture (routers -> logic -> db -> core), we try to follow a n-layer architecture approach, where each layer has a single responsibility. I think the DB layer should only do the first one and should be generic and simple.

See https://diracx.diracgrid.org/en/latest/dev/explanations/components/#python-modules

I would likely have 2 methods here: one to query the sites and one to query the resources.
Then the logic would call one of these 2 methods, and would return the right pydantic model based on the result (so it would likely have 4 functions: site, storage, compute, fts).

What do you think?

You're right, I think that's actually better. I've organized the features according to the diracx's architecture.

diracx-db/pyproject.toml

aldbr · 2026-04-09T15:50:05Z

diracx-logic/src/diracx/logic/rss/rss.py

What about calling that file query.py?

aldbr · 2026-04-09T15:52:28Z

diracx-core/src/diracx/core/models/rss.py

+def map_status(db_status: str, reason: str | None = None) -> ResourceStatus:
+    if db_status in ALLOWED:
+        return AllowedStatus(allowed=True)
+
+    if db_status in BANNED:
+        return BannedStatus(
+            allowed=False,
+            reason=reason or db_status,
+        )
+
+    return BannedStatus(
+        allowed=False,
+        reason=f"Unknown status: {db_status}",
+    )


What about moving map_status to diracx-logic?

aldbr · 2026-04-09T15:53:20Z

diracx-core/tests/test_rss.py

And therefore you can potentially move that test to diracx-logic too

aldbr · 2026-04-09T16:20:56Z

diracx-db/src/diracx/db/sql/rss/schema.py

+
+
+class ElementStatusBase:
+    __table_args__ = {"mysql_engine": "InnoDB", "mysql_charset": "utf8mb4"}


Do you really need that?

Suggested change

__table_args__ = {"mysql_engine": "InnoDB", "mysql_charset": "utf8mb4"}

aldbr · 2026-04-09T16:23:57Z

diracx-db/src/diracx/db/sql/rss/schema.py

+    vo: Mapped[str64] = mapped_column("VO", primary_key=True, server_default="all")
+    status: Mapped[str] = mapped_column("Status", String(8), server_default="")
+    reason: Mapped[str512] = mapped_column("Reason", server_default="Unspecified")
+    dateeffective: Mapped[datetime] = mapped_column("DateEffective", DateTime)


Don't you need to pass SmarterDateTime here? Same for the other dates.

Suggested change

dateeffective: Mapped[datetime] = mapped_column("DateEffective", DateTime)

date_effective: Mapped[datetime] = mapped_column("DateEffective", SmarterDateTime())

aldbr · 2026-04-09T16:24:12Z

diracx-db/src/diracx/db/sql/rss/schema.py

+        "ID", BigInteger, autoincrement=True, primary_key=True
+    )
+    name: Mapped[str64] = mapped_column("Name")
+    statustype: Mapped[str128] = mapped_column("StatusType", server_default="all")


Any reason for not using snake case as in the other table definitions? Same for all the attribute names

Suggested change

statustype: Mapped[str128] = mapped_column("StatusType", server_default="all")

status_type: Mapped[str128] = mapped_column("StatusType", server_default="all")

aldbr · 2026-04-09T16:28:15Z

diracx-core/src/diracx/core/models/rss.py

+
+def map_status(db_status: str, reason: str | None = None) -> ResourceStatus:
+    if db_status in ALLOWED:
+        return AllowedStatus(allowed=True)


What about the warnings?

The should be what is in reason, at least for the case when db_status is "Degraded".

aldbr · 2026-04-09T16:29:45Z

diracx-db/tests/rss/test_rss_db.py

+        )
+
+    # Test with the test Site (should be found)
+    async with rss_db as rss_db:


Minor: to avoid name shadowing

Suggested change

async with rss_db as rss_db:

async with rss_db as db:

aldbr · 2026-04-09T16:31:57Z

diracx-db/src/diracx/db/sql/rss/db.py

+
+    metadata = RSSBase.metadata
+
+    async def get_site_status(self, name: str, vo: str = "all") -> tuple[str, str]:


@fstagni out of curiosity, shouldn't we (not now but later) transition from this magic all to NULL in the DB?
If not VO is set, then we assume it's for all of them.
Unless I miss something?

I would agree with you

aldbr · 2026-04-09T16:34:40Z

diracx-db/src/diracx/db/sql/rss/db.py

+    async def get_resource_status(
+        self,
+        name: str,
+        statustypes: list[str] = ["all"],


Beware of https://docs.astral.sh/ruff/rules/mutable-argument-default/
It's fine here because we don't mutate status_types but we should be careful.

@fstagni same here, should we transition from all for VO and StatusType to NULL at some point?

I would agree to this one too.

fstagni · 2026-04-10T08:55:03Z

diracx-core/src/diracx/core/models/rss.py

+
+def map_status(db_status: str, reason: str | None = None) -> ResourceStatus:
+    if db_status in ALLOWED:
+        return AllowedStatus(allowed=True)


The should be what is in reason, at least for the case when db_status is "Degraded".

fstagni · 2026-04-10T08:57:40Z

diracx-db/src/diracx/db/sql/rss/db.py

+
+    metadata = RSSBase.metadata
+
+    async def get_site_status(self, name: str, vo: str = "all") -> tuple[str, str]:


I would agree with you

fstagni · 2026-04-10T08:58:04Z

diracx-db/src/diracx/db/sql/rss/db.py

+    async def get_resource_status(
+        self,
+        name: str,
+        statustypes: list[str] = ["all"],


I would agree to this one too.

fstagni · 2026-04-10T08:59:56Z

diracx-core/src/diracx/core/exceptions.py

        )


+class ResourceNotFoundError(DiracError):


I am really wondering if we need this one at all.

HeloiseJoffe requested a review from aldbr March 25, 2026 09:33

fstagni reviewed Mar 25, 2026

View reviewed changes

fstagni closed this Mar 26, 2026

fstagni reopened this Mar 26, 2026

aldbr force-pushed the feat/rss-data-models branch from dee80c6 to bf5abcb Compare March 26, 2026 18:26

HeloiseJoffe force-pushed the feat/rss-data-models branch from 59514d8 to 036cb9c Compare March 30, 2026 08:35

aldbr requested changes Mar 31, 2026

View reviewed changes

DIRACGridBot marked this pull request as draft March 31, 2026 08:59

HeloiseJoffe force-pushed the feat/rss-data-models branch 2 times, most recently from cb53ac5 to 87c7b68 Compare April 2, 2026 06:37

HeloiseJoffe marked this pull request as ready for review April 2, 2026 06:39

HeloiseJoffe added 4 commits April 8, 2026 11:06

feat: add RSS data models (SQLAlchemy + Pydantic)

ac2ecbe

fix: remove deprecated tables and add FTS ResourceType

9dde579

refactor: restructure to follow n-layer architecture

e31edfe

chore: remove unnecessary comments

5c3f59d

HeloiseJoffe force-pushed the feat/rss-data-models branch from 87c7b68 to 5c3f59d Compare April 8, 2026 09:29

HeloiseJoffe requested a review from aldbr April 8, 2026 12:06

aldbr requested changes Apr 9, 2026

View reviewed changes

DIRACGridBot marked this pull request as draft April 9, 2026 16:36

fstagni reviewed Apr 10, 2026

View reviewed changes



		class ElementStatusBase:
		__table_args__ = {"mysql_engine": "InnoDB", "mysql_charset": "utf8mb4"}

	dateeffective: Mapped[datetime] = mapped_column("DateEffective", DateTime)
	date_effective: Mapped[datetime] = mapped_column("DateEffective", SmarterDateTime())

	statustype: Mapped[str128] = mapped_column("StatusType", server_default="all")
	status_type: Mapped[str128] = mapped_column("StatusType", server_default="all")


		metadata = RSSBase.metadata

		async def get_site_status(self, name: str, vo: str = "all") -> tuple[str, str]:

Conversation

HeloiseJoffe commented Mar 25, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

read-the-docs-community bot commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation build overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

read-the-docs-community bot commented Mar 26, 2026 •

edited

Loading