Skip to content

[DPE-9049] feat: implement automatic cleanup of lost+found storage directories#1612

Open
marceloneppel wants to merge 1 commit into16/edgefrom
remove-lost+found
Open

[DPE-9049] feat: implement automatic cleanup of lost+found storage directories#1612
marceloneppel wants to merge 1 commit into16/edgefrom
remove-lost+found

Conversation

@marceloneppel
Copy link
Copy Markdown
Member

@marceloneppel marceloneppel commented Apr 13, 2026

Issue

When deploying Charmed PostgreSQL with a dedicated storage volume (e.g., NVMe drives formatted with ext4), the filesystem automatically creates a lost+found directory at the root of the mount point. initdb treats the storage directory as non-empty and refuses to initialise the database cluster, resulting in an error such as:

initdb: error: directory "/var/snap/charmed-postgresql/common/var/lib/postgresql" exists but is not empty
initdb: detail: It contains a lost+found directory, perhaps due to it being a mount point.
initdb: hint: Using a mount point directly as the data directory is not recommended.

This prevents the PostgreSQL cluster from ever forming without manual intervention.

Solution

The charm now automatically removes any lost+found directory found at the root of each storage path (data, archive, logs, and temp) during the install and start events, eliminating the need for external workarounds.

Tested with ext4 volumes in a Testflinger machine (it's needed to use LXD VMs instead of containers for it to work).

Checklist

  • I have added or updated any relevant documentation.
  • I have cleaned any remaining cloud resources from my accounts.

Fixes #1336.

The `lost+found` directory, automatically generated by certain storage
substrates (like LXD block-pool volumes), can interfere with PostgreSQL's
initialization and operations by introducing unexpected files in the data roots.

Signed-off-by: Marcelo Henrique Neppel <marcelo.neppel@canonical.com>
@github-actions github-actions bot added the Libraries: OK The charm libs used are OK and in-sync label Apr 13, 2026
@marceloneppel marceloneppel added the bug Something isn't working as expected label Apr 13, 2026
@marceloneppel marceloneppel changed the title feat: implement automatic cleanup of lost+found storage directories [DPE-9049] feat: implement automatic cleanup of lost+found storage directories Apr 13, 2026
@marceloneppel marceloneppel marked this pull request as ready for review April 13, 2026 18:05
@marceloneppel marceloneppel requested a review from a team as a code owner April 13, 2026 18:05
@marceloneppel marceloneppel requested review from carlcsaposs-canonical, dragomirp, juju-charm-bot and taurus-forever and removed request for a team April 13, 2026 18:05
Comment thread src/charm.py
shutil.rmtree(lost_and_found_path)
except OSError:
logger.exception(f"Failed to remove {lost_and_found_path}")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the history, we cannot rename/move even the PostgreSQL does ccare about dot_files and lost+found here: https://github.com/postgres/postgres/blob/66ad764c8d517f59577d41ac3dad786729c9e10e/src/port/pgcheckdir.c#L60

it still aborts basebackup execution in https://github.com/postgres/postgres/blob/66ad764c8d517f59577d41ac3dad786729c9e10e/src/bin/pg_basebackup/pg_basebackup.c#L771-L772

This is a new Juju volume which obviously has empty lost+found folder, so we are cleaning it to have deployment fixed. The folder will be re-created automatically.

@taurus-forever taurus-forever requested a review from delgod April 14, 2026 12:21
Copy link
Copy Markdown
Member

@delgod delgod left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can be smarter here, clean up the directory only if initdb is run nessesary.
The problem right now is that if we have a healthy PostgreSQL and the kernel crashes, and the machine is restarted, it can get a valid lost+found folder needed for recovery.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working as expected Libraries: OK The charm libs used are OK and in-sync

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants