ImageProperty and VideoProperty not working

[benfrancis]

STR:

• Add virtual VideoCamera Thing
• Navigate to detail view
• Try to watch video

Expected:

• Can watch video

Actual:

• Video not playable

I haven't tested this with real video camera hardware recently.

I've noticed that the video property affordance looks wrong in the Thing Description. At least one of the two forms should have a video content type but instead one has sse as a subprotocol (which is not useful for a video stream).

This could potentially be a regression from the implementation of the HTTP SSE Profile, or possibly just a bug in the Virtual Things add-on.

[GHoGit]

What is the expected output from this virtual player ?
I can see the player controls, but only if the cursor is hovering the controls area.
If some dummy video is supposed to show, I see nothing.

Is this in some way releated to behaviour of the ONVIF add-on ?
I have several TP-Link Tapo cameras that I can view with VLC using RTSP protocol, but never had any luck with ONVIF.
That being said, the adapter docs do not claim compliance with Tapo.
From the logs I see that camera info is read, but apparently not accepted by the add-on.
This of course applies to WT1.1.

[benfrancis]

@GHoGit wrote:

What is the expected output from this virtual player ?

In the case of the Virtual Things add-on the demo video camera device is meant to stream a video from a static video file (it's a movie trailer).

Is this in some way releated to behaviour of the ONVIF add-on ?

Yes, based on my findings below, I think that add-on is likely to be broken too.

I've noticed that both ImageProperty and VideoProperty are not being represented correctly in Thing Descriptions:

ImageProperty:

      "image": {
        "name": "image",
        "value": null,
        "title": "Image",
        "type": "null",
        "@type": "ImageProperty",
        "readOnly": true,
        "forms": [
          {
            "href": "/things/virtual-things-19/properties/image"
          },
          {
            "href": "/things/virtual-things-19/properties/image",
            "op": [
              "observeproperty",
              "unobserveproperty"
            ],
            "subprotocol": "sse"
          }
        ]
      }

VideoProperty:

      "video": {
        "name": "video",
        "value": null,
        "title": "Video",
        "type": "null",
        "@type": "VideoProperty",
        "readOnly": true,
        "forms": [
          {
            "href": "/things/virtual-things-20/properties/video"
          },
          {
            "href": "/things/virtual-things-20/properties/video",
            "op": [
              "observeproperty",
              "unobserveproperty"
            ],
            "subprotocol": "sse"
          }
        ]
      }

What used to happen before all the W3C compliance changes was that add-ons would provide links to image and video resources to be included in property metadata in the Thing Description:

ImageProperty:

{
      name: 'image',
      value: null,
      metadata: {
        type: 'null',
        '@type': 'ImageProperty',
        title: 'Image',
        readOnly: true,
        links: [
          {
            rel: 'alternate',
            href: '/media/virtual-things/image.png',
            mediaType: 'image/png',
          },
        ],
      },
    }
}

VideoProperty:

{
        type: 'null',
        '@type': 'VideoProperty',
        title: 'Video',
        readOnly: true,
        links: [
          {
            rel: 'alternate',
            href: '/media/virtual-things/index.mpd',
            mediaType: 'application/dash+xml',
          },
        ],
      }
}

It appears that since the changes this custom metadata is now ignored and instead generic forms are added which imply that the image and video properties have a JSON representation which can also be subscribed to using Server-Sent Events, which is clearly not true.

I think we need to allow add-ons to supply custom Forms instead of Links and update the add-ons accordingly. This has always been difficult to express in a W3C compliant way, but I'm thinking something like:

ImageProperty:

      "image": {
        "name": "image",
        "title": "Image",
        "type": "null",
        "@type": "ImageProperty",
        "readOnly": true,
        "forms": [
          {
            "op": "readproperty",
            "href": "/media/virtual-things/image.png",
            "contentType": "image/png",
          }
        ]
      }

VideoProperty:

      "video": {
        "name": "video",
        "title": "Video",
        "type": "null",
        "@type": "VideoProperty",
        "readOnly": true,
        "forms": [
          {
            "op": "readproperty",
            "href": "/media/virtual-things/index.mpd",
            "contentType": "application/dash+xml",
          }
        ]
      }

[benfrancis]

I've noticed that @relu91 actually diligently added tests for this functionality in #2811, so it's possible this actually already works in the gateway and it's just the add-ons that need updating to specify forms rather than links.

Edit: See also, #3091, of which this is probably a duplicate but I'll close that one since this one provides more information. And WebThingsIO/virtual-things-adapter#68 in which I already noticed that the virtual things adapter needed updating 🤦‍♂️

w3c/wot-thing-description#1814 may also be a problem for meta operations, so we should test that.

[benfrancis]

I have verified that this does actually work and I have submitted a PR to update the virtual things add-on at WebThingsIO/virtual-things-adapter#71

There's some follow-up work to do in #3230 which applies to Thing Descriptions more generally, and some add-ons will need to be updated, but I'm going to close this issue since it appears ImageProperty and VideoProperty do actually work.