[Bug]: File uploads intermittently fail with "Object not found after writing" due to object storage race condition

[tirenucci]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

I have encountered a bug where our Nextcloud server cannot manage to upload files, in a normal or team folder:

• File uploads fail with "Object not found after writing" error due to a race condition in object storage validation.

Steps to reproduce

1. Use Nextcloud with object storage backend (S3/Swift/etc.)
2. Upload files
3. Error occurs intermittently due to timing
4. The timing-dependent nature makes it difficult to reproduce consistently.

Expected behavior

Retry mechanism & File uploads should succeed

Nextcloud Server version

32

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.3

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Upgraded to a MAJOR version (ex. 31 to 32)

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

{
    "system": {
        "htaccess.RewriteBase": "\/",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "overwriteprotocol": "https",
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "objectstore": {
            "class": "\\OC\\Files\\ObjectStore\\S3",
            "arguments": {
                "bucket": "fneaurabuckets3",
                "region": "sbg",
                "hostname": "s3.sbg.io.cloud.ovh.net",
                "port": "443",
                "storageClass": "",
                "objectPrefix": "urn:oid:",
                "autocreate": true,
                "use_ssl": true,
                "use_path_style": true,
                "legacy_auth": false,
                "key": "***REMOVED SENSITIVE VALUE***",
                "secret": "***REMOVED SENSITIVE VALUE***"
            }
        },
        "upgrade.disable-web": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "intranet.next.fne-aura.org"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "32.0.0.13",
        "overwrite.cli.url": "https:\/\/intranet.next.fne-aura.org",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "default_phone_region": "FR",
        "enable_previews": false,
        "maintenance_window_start": 1,
        "default_language": "fr",
        "default_locale": "fr_FR",
        "default_timezone": "Europe\/Paris",
        "auto_logout": true,
        "knowledgebaseenabled": false,
        "skeletondirectory": "",
        "mail_smtpmode": "smtp",
        "mail_smtpsecure": "ssl",
        "mail_sendmailmode": "smtp",
        "mail_smtpauth": 1,
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "loglevel": 0,
        "force_language": "fr",
        "force_locale": "fr_FR",
        "trashbin_retention_obligation": "auto, 90",
        "versions_retention_obligation": "disabled",
        "memories.exiftool": "\/var\/www\/html\/custom_apps\/memories\/bin-ext\/exiftool-amd64-glibc",
        "memories.vod.path": "\/var\/www\/html\/custom_apps\/memories\/bin-ext\/go-vod-amd64",
        "memories.vod.ffmpeg": "\/usr\/bin\/ffmpeg",
        "memories.vod.ffprobe": "\/usr\/bin\/ffprobe",
        "app_install_overwrite": [
            "libresign"
        ],
        "maintenance": false,
        "defaultapp": "dashboard,files,activity"
    }
}

List of activated Apps

Enabled:
  - activity: 5.0.0-dev.0
  - app_api: 32.0.0
  - bruteforcesettings: 5.0.0-dev.0
  - calendar: 6.0.1
  - circles: 32.0.0
  - cloud_federation_api: 1.16.0
  - comments: 1.22.0
  - contacts: 8.0.3
  - contactsinteraction: 1.13.1
  - dashboard: 7.12.0
  - dav: 1.34.2
  - deck: 1.16.0
  - federatedfilesharing: 1.22.0
  - federation: 1.22.0
  - files: 2.4.0
  - files_downloadlimit: 5.0.0-dev.0
  - files_pdfviewer: 5.0.0-dev.0
  - files_reminders: 1.5.0
  - files_sharing: 1.24.0
  - files_trashbin: 1.22.0
  - files_versions_s3: 1.3.0
  - firstrunwizard: 5.0.0-dev.0
  - forms: 5.2.1
  - impersonate: 3.0.0
  - libresign: 11.4.1
  - logreader: 5.0.0-dev.0
  - lookup_server_connector: 1.20.0
  - mail: 5.5.7
  - nextcloud_announcements: 4.0.0-dev.0
  - notes: 4.12.3
  - notifications: 5.0.0-dev.0
  - notify_push: 1.2.0
  - oauth2: 1.20.0
  - password_policy: 4.0.0-dev.0
  - polls: 8.5.0
  - privacy: 4.0.0-dev.0
  - profile: 1.1.0
  - provisioning_api: 1.22.0
  - recommendations: 5.0.0-dev.0
  - related_resources: 3.0.0-dev.0
  - richdocuments: 9.0.0
  - serverinfo: 4.0.0-dev.0
  - settings: 1.15.1
  - sharebymail: 1.22.0
  - spreed: 22.0.0
  - survey_client: 4.0.0-dev.0
  - systemtags: 1.22.0
  - text: 6.0.0-dev.0
  - theming: 2.7.0
  - twofactor_backupcodes: 1.21.0
  - updatenotification: 1.22.0
  - user_status: 1.12.0
  - viewer: 5.0.0-dev.0
  - weather_status: 1.12.0
  - webhook_listeners: 1.3.0
  - workflowengine: 2.14.0
Disabled:
  - admin_audit: 1.22.0 (installed 1.21.0)
  - collectives: 3.2.0 (installed 3.2.0)
  - encryption: 2.20.0 (installed 2.19.0)
  - files_external: 1.24.0 (installed 1.24.0)
  - files_versions: 1.25.0 (installed 1.24.0)
  - photos: 5.0.0-dev.1 (installed 2.5.0)
  - support: 4.0.0-dev.0 (installed 3.0.0)
  - suspicious_login: 10.0.0-dev.0
  - twofactor_nextcloud_notification: 6.0.0-dev.0
  - twofactor_totp: 14.0.0
  - user_ldap: 1.23.0 (installed 1.22.0)

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

Additional info

We use S3 bucket provided by OVH. Maybe not fast as aws s3.

In the upload code, objectExists() verification happens immediately after writing, but object storage has latency.

Maybe add retry mechanism or make validation optional for object storage backends.

see ->
https://github.com/nextcloud/server/blob/8d2a95678fd1db0dda0567efadb2d3d667aaa689/lib/private/Files/ObjectStore/ObjectStoreStorage.php#L556C3-L567C4

I've temporarily resolved this issue by removing/disabling the objectExists() validation check:

[joshtrichards]

If understanding correctly, you should be able to the following in your config.php or equivalent:

'objectstore' => [
        'class' => '\\OC\\Files\\ObjectStore\\S3',
        'arguments' => [
        [...]
                `'validateWrites' => false`
        [...]

This was added back for this type of situation back in PR #33458, but doesn't look like it got documented.

[tirenucci]

Okay sound's good ! Thank's :)

[johackim]

I have the same issue using this Nextcloud config with OVH too.

An OBJECTSTORE_S3_VALIDATE_WRITES environment variable would be a great! 😃

[as598466]

Same problem with OVH lately. Resolved by using the variable 'validateWrites' => false. The documentation really needs updating.