fix: prevent uint32_t overflow in avifSetTileConfiguration tile area

[uwezkhan]

fix: prevent uint32_t overflow in avifSetTileConfiguration tile area

When computing the tile count, width * height was performed as a
uint32_t multiplication. For images with dimensions whose product
exceeds UINT32_MAX (e.g. 100000x50000), this silently wraps around,
producing an incorrect tile count and potentially corrupt tile layout.

Fix by widening to uint64_t before multiplying, and clamping to
kMaxTiles using AVIF_MIN before downcasting back to uint32_t.
The existing bounds checks are preserved as a safety net.

Fixes: integer overflow in avifSetTileConfiguration (src/write.c)

[uwezkhan]

Thanks for the clarification. I’ve updated the implementation to use uint64_t for imageArea, safely cast the result back to uint32_t, and verified that the caller enforces width and height ≤ 65536.

[wantehchang]

uwezkhan: You wrote:

I’ve updated the implementation to use uint64_t for imageArea, safely cast the result back to uint32_t, and verified that the caller enforces width and height ≤ 65536.

Could you tell me where the caller of avifSetTileConfiguration() enforces width and height ≤ 65536?

I can't seem to find that code. I just wrote PR #3061 to do that.

[wantehchang]

I verified with the following patch that the caller of avifSetTileConfiguration() does not enforce width and height ≤ 65536. So we need something like PR #3061.

diff --git a/src/write.c b/src/write.c
index f86bdd45..79b68e30 100644
--- a/src/write.c
+++ b/src/write.c
@@ -5,6 +5,7 @@
 
 #include <assert.h>
 #include <stdint.h>
+#include <stdio.h>
 #include <string.h>
 #include <time.h>
 
@@ -88,6 +89,7 @@ static void splitTilesLog2(uint32_t dim1, uint32_t dim2, int tilesLog2, int * ti
 // dimension of the image to make the tile size closer to a square.
 void avifSetTileConfiguration(int threads, uint32_t width, uint32_t height, int * tileRowsLog2, int * tileColsLog2)
 {
+    fprintf(stderr, "avifSetTileConfiguration: width=%u height=%u\n", width, height);
     *tileRowsLog2 = 0;
     *tileColsLog2 = 0;
     if (threads > 1) {
diff --git a/tests/gtest/avifallocationtest.cc b/tests/gtest/avifallocationtest.cc
index c4ad7cd1..1b3d0292 100644
--- a/tests/gtest/avifallocationtest.cc
+++ b/tests/gtest/avifallocationtest.cc
@@ -138,6 +138,7 @@ void TestEncoding(uint32_t width, uint32_t height, uint32_t depth,
   EncoderPtr encoder(avifEncoderCreate());
   ASSERT_NE(encoder, nullptr);
   encoder->speed = AVIF_SPEED_FASTEST;
+  encoder->autoTiling = AVIF_TRUE;
   testutil::AvifRwData encoded_avif;
   ASSERT_EQ(avifEncoderWrite(encoder.get(), image.get(), &encoded_avif),
             expected_result);

[wantehchang]

I merged PR #3061, so the caller now validates width <= 65536 and height <= 65536.

Note: For the purpose of this function, we could also clamp width and height to 65536 within this function. Then we don't need the caller to validate that.