BIO improvements and fixes

[rizlik]

Description

• simplify wolfSSL_BIO_set_conn_hostname, fixing OOB read
• restructure wolfSSL_BIO_ctrl_pending, fixing inverted check and
• ctrlCB checking
• return WOLFSSL_FAILURE in wolfSSL_BIO_up_ref when refInc fails,
  updated test to reflect this
• check arguments for NULL in wolfSSL_BIO_ADDR_size
• replace non-portable type long usigned int with size_t
• wolfSSL_BIO_MEMORY_write: return WOLFSSL_BIO_ERROR on failure instead
  of WOLFSSL_FAILURE, return 0 when len is 0
• wolfSSL_BIO_get_fp: fix type mismatch comparing XFILE* pointer against
  XBADFILE
• wolfSSL_BIO_ctrl: add NULL check on bio before switch
• wolfSSL_BIO_pop: clear bio prev and next pointers after unlinking
• wolfSSL_BIO_gets: place null terminator after actual bytes read from
  BIO_BIO nread

Fixes zd#21528

[Copilot]

Pull request overview

This PR improves wolfSSL’s OpenSSL-compat BIO layer by tightening argument validation, correcting return/error semantics, fixing a reported OOB read, and adding targeted regression tests (zd#21528).

Changes:

• Fixes/reshapes several BIO behaviors (ctrl/ctrl_pending traversal, BIO_gets NUL termination, BIO_pop unlink safety, BIO_up_ref failure handling, BIO_get_fp argument checking).
• Adjusts MEM BIO write semantics to return WOLFSSL_BIO_ERROR on failure and 0 on zero-length writes; updates tests and adds new regression tests for hostname and pending traversal.
• Adds a BIO method type upper bound (WOLFSSL_MAX_BIO_TYPE) and enforces it in wolfSSL_BIO_meth_new().

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
wolfssl/ssl.h	Introduces WOLFSSL_MAX_BIO_TYPE to bound custom BIO method types.
tests/api/test_ossl_bio.h	Registers two new BIO regression tests in the test declaration list.
tests/api/test_ossl_bio.c	Updates an existing expectation for MEM BIO write failure and adds new tests for BIO_set_conn_hostname and chained BIO_ctrl_pending.
src/bio.c	Implements the BIO fixes/improvements across MEM BIO write, ctrl/ctrl_pending, gets termination, addr size NULL handling, get_fp arg validation, meth_new type bounds, conn hostname allocation logic, and pop unlink cleanup.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[dgarske]

🐺 Skoll Code Review

Overall recommendation: APPROVE
Findings: 6 total — 2 posted, 4 skipped

Posted findings

• [Info] Stale comment still references WOLFSSL_FAILURE in wolfSSL_BIO_MEMORY_write — src/bio.c:595-596
• [Medium] wolfSSL_BIO_ctrl_pending: ctrlCb check could dispatch to callback for built-in wrapper BIO types in chain — src/bio.c:1308-1314

Skipped findings

• [Medium] test_wolfSSL_BIO_set_conn_hostname does not verify stored hostname value
• [Medium] No test coverage for wolfSSL_BIO_pop prev/next clearing
• [Medium] No test for wolfSSL_BIO_up_ref failure return when WOLFSSL_REFCNT_ERROR_RETURN is defined
• [Low] wolfSSL_BIO_MEMORY_write returns 0 for bio==NULL but callers may not expect this

Review generated by Skoll via openclaw