PS-10068 Adding new KMIP C++ library and making it production ready

[lukin-oleksiy]

https://perconadev.atlassian.net/browse/PS-10068

The new kmipclient added to replacre kmippp; the new kmipcore library added to replace legacy libkmip; This is AI assited code based on initial implementation:

PS-9697 new KMIP C++ client library that replaces midle and top levels of old one

For more information please see KMIP_MODERN_VS_LEGACY_COMPARISON.md file

[Copilot]

Pull request overview

This PR introduces a new “modern” KMIP stack by adding the kmipcore C++ protocol/TTLV layer and the kmipclient C++20 high-level client layer, intended to replace the legacy libkmip + kmippp stack.

Changes:

• Added kmipcore library (TTLV model, serialization buffer, request/response/payload types, parsers, logging/formatting hooks).
• Added kmipclient library (OpenSSL transport, high-level KMIP operations, connection pool, examples, integration-test scaffolding).
• Updated top-level build wiring and added docs/comparison material (Doxygen target + comparison doc).

Reviewed changes

Copilot reviewed 73 out of 73 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
CMakeLists.txt	Wires in new subprojects + Doxygen/doc target.
Doxyfile	Adds Doxygen configuration.
KMIP_MODERN_VS_LEGACY_COMPARISON.md	Documents modern vs legacy stack differences.
kmipcore/CMakeLists.txt	Builds/install kmipcore and test executables.
kmipcore/.clang-format	Adds formatting rules for kmipcore.
kmipcore/include/kmipcore/attributes_parser.hpp	Declares attribute decoding helper.
kmipcore/include/kmipcore/key.hpp	Defines core Key model.
kmipcore/include/kmipcore/key_parser.hpp	Declares typed Get-response parsing into Key/Secret.
kmipcore/include/kmipcore/kmip_attribute_names.hpp	Centralizes attribute-name constants.
kmipcore/include/kmipcore/kmip_basics.hpp	Defines core TTLV Element model + KmipException.
kmipcore/include/kmipcore/kmip_formatter.hpp	Declares human-readable formatting API.
kmipcore/include/kmipcore/kmip_logger.hpp	Declares logging interface + NullLogger.
kmipcore/include/kmipcore/kmip_requests.hpp	Adds typed request batch item wrappers.
kmipcore/include/kmipcore/kmip_responses.hpp	Adds typed response batch item wrappers.
kmipcore/include/kmipcore/kmipcore_version.hpp	Introduces kmipcore version macros.
kmipcore/include/kmipcore/response_parser.hpp	Adds ResponseParser API for typed access to batch items.
kmipcore/include/kmipcore/secret.hpp	Defines core Secret model.
kmipcore/include/kmipcore/serialization_buffer.hpp	Adds reusable TTLV serialization buffer abstraction.
kmipcore/include/kmipcore/types.hpp	Adds common typedefs and state string helpers.
kmipcore/src/attributes_parser.cpp	Implements attribute element decoding.
kmipcore/src/key_parser.cpp	Implements Get-response parsing into Key/Secret.
kmipcore/src/kmip_payloads.cpp	Implements Locate payload encode/decode + Attribute model.
kmipcore/src/kmip_requests.cpp	Implements higher-level request builders (create/register/locate/revoke).
kmipcore/src/kmip_responses.cpp	Implements typed response wrappers parsing.
kmipcore/src/response_parser.cpp	Implements response parsing + operation/result formatting.
kmipcore/src/serialization_buffer.cpp	Implements SerializationBuffer growth/padding/release.
kmipcore/tests/test_core.cpp	Adds basic serialization/deserialization sanity tests.
kmipcore/tests/test_serialization_buffer.cpp	Adds SerializationBuffer behavior tests.
kmipclient/.clang-format	Adds formatting rules for kmipclient.
kmipclient/CHANGELOG.md	Introduces kmipclient changelog.
kmipclient/CMakeLists.txt	Builds kmipclient + examples + optional gtest integration tests.
kmipclient/README.md	Documents kmipclient API, design, build, and examples.
kmipclient/TODO.md	Tracks remaining feature work.
kmipclient/examples/example_activate.cpp	Adds activate example program.
kmipclient/examples/example_create_aes.cpp	Adds AES create example program.
kmipclient/examples/example_destroy.cpp	Adds destroy example program.
kmipclient/examples/example_get.cpp	Adds get-key example program.
kmipclient/examples/example_get_all_ids.cpp	Adds list-all-ids example program.
kmipclient/examples/example_get_attributes.cpp	Adds get-attributes example program.
kmipclient/examples/example_get_logger.cpp	Adds get-key-with-logger example program.
kmipclient/examples/example_get_name.cpp	Adds get-name/group example program.
kmipclient/examples/example_get_secret.cpp	Adds get-secret example program.
kmipclient/examples/example_locate.cpp	Adds locate-by-name example program.
kmipclient/examples/example_locate_by_group.cpp	Adds locate-by-group example program.
kmipclient/examples/example_pool.cpp	Adds multi-threaded pool example program.
kmipclient/examples/example_register_key.cpp	Adds register-key example program.
kmipclient/examples/example_register_secret.cpp	Adds register-secret example program.
kmipclient/examples/example_revoke.cpp	Adds revoke example program.
kmipclient/include/kmipclient/Kmip.hpp	Adds facade that owns transport+client.
kmipclient/include/kmipclient/KmipClient.hpp	Declares high-level KMIP operations API.
kmipclient/include/kmipclient/KmipClientPool.hpp	Declares thread-safe connection pool API.
kmipclient/include/kmipclient/KmipIOException.hpp	Declares IO exception type.
kmipclient/include/kmipclient/Key.hpp	Declares client-level Key helpers (hex/base64/PEM/generate).
kmipclient/include/kmipclient/NetClient.hpp	Declares abstract transport interface.
kmipclient/include/kmipclient/NetClientOpenSSL.hpp	Declares OpenSSL BIO transport.
kmipclient/include/kmipclient/kmipclient_version.hpp	Introduces kmipclient version macros.
kmipclient/include/kmipclient/types.hpp	Re-exports kmipcore types into kmipclient.
kmipclient/src/IOUtils.cpp	Implements KMIP framing send/receive + debug logging.
kmipclient/src/IOUtils.hpp	Declares IO helper for request/response exchange.
kmipclient/src/Key.cpp	Implements PEM parsing + AES key utilities.
kmipclient/src/KmipClient.cpp	Implements high-level operations via kmipcore requests/parsers.
kmipclient/src/KmipClientPool.cpp	Implements thread-safe pool with RAII borrowed clients.
kmipclient/src/NetClientOpenSSL.cpp	Implements OpenSSL BIO TLS transport with timeouts.
kmipclient/src/StringUtils.cpp	Implements hex and base64 decoding helpers.
kmipclient/src/StringUtils.hpp	Declares string/encoding helper utilities.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[inikep]

Review of 103e598

Findings

High

1. kmipclient/src/NetClientOpenSSL.cpp - NetClientOpenSSL::connect() loads the CA file but never enables peer verification and never verifies the hostname.

   The code creates the TLS context, loads trust anchors with SSL_CTX_load_verify_locations(), and connects with BIO_do_connect(), but it never calls SSL_CTX_set_verify(..., SSL_VERIFY_PEER, ...), SSL_set1_host(), or equivalent hostname/SAN checks. In OpenSSL client mode that means the library can establish a TLS session without actually authenticating the KMIP server, which is a man-in-the-middle risk.

2. kmipcore/include/kmipcore/kmip_enums.hpp, kmipcore/include/kmipcore/kmip_basics.hpp, kmipcore/src/kmip_basics.cpp - the public enum exposes KMIP_TYPE_DATE_TIME_EXTENDED, but the TTLV core cannot serialize or deserialize it.

   KMIP_TYPE_DATE_TIME_EXTENDED is defined in the enums and even recognized by the formatter, but Value has no matching variant and Element::deserialize() falls through to Unknown type for 0x0B. Any KMIP 2.0 message using that TTLV type will fail even though the new API surface suggests it is supported.

Medium

3. kmipcore/src/kmip_basics.cpp - Element::deserialize() does not enforce structure body boundaries.

   In the KMIP_TYPE_STRUCTURE branch, children are parsed from the full remaining buffer until current_struct_offset < length becomes false, but there is no check that each child fits within the remaining structure body and no final current_struct_offset == length validation. A malformed child can consume bytes past the declared structure length and the parser will still return success, effectively swallowing sibling data into the structure.

4. kmipcore/src/kmip_protocol.cpp - malformed responses are accepted because required response fields are not enforced.

   ResponseMessage::fromElement() accepts a response with no ResponseHeader at all, leaving header_ default-initialized. ResponseBatchItem::fromElement() also accepts a batch item with no Operation, leaving operation_ as 0. The request-side parser rejects the symmetric errors, so this asymmetry means corrupted or hostile responses can be treated as valid and only fail later with generic "Unknown" operation handling.

5. kmipcore/include/kmipcore/kmip_protocol.hpp, kmipcore/src/kmip_protocol.cpp - RequestHeader exposes username/password setters that are silently ignored on the wire.

   The class has setUserName() / setPassword() and matching getters, but RequestHeader::toElement() and RequestHeader::fromElement() never encode or decode any authentication fields. A caller can populate credentials through the public API and still send an unauthenticated request, which is a correctness bug in the exposed interface.

6. kmipclient/include/kmipclient/KmipClient.hpp, kmipclient/src/KmipClient.cpp - op_locate_by_group() and op_all() can return more than max_ids.

   The header promises "up to max_ids entries", but the implementation appends a full locate page before checking result.size() < max_ids in the loop condition. If max_ids is smaller than one page, or the last page crosses the remaining budget, the function returns more IDs than requested.

Low

7. CMakeLists.txt - top-level configuration unconditionally requires Doxygen.

   find_package(Doxygen REQUIRED) makes configuration fail on systems without Doxygen, while the following else() branch claims documentation generation will be skipped when Doxygen is absent. That branch is unreachable with REQUIRED, so the new top-level build setup is internally inconsistent and unnecessarily breaks non-doc users.

[lukin-oleksiy]

All Copilot findings fixed; Review of 103e598 fixes:

1. We do not check any certificates signature because almost all users of the library use self-signed certificates or some signed by corporate. Also, we do not verify the host name because DNS is not always work properly in a cloud environment. So, this is by design.
   2, 3, 4 fixed;
2. We do not have use cases with user authentication yet, but feature is implemented to have complete API. This feature is experimental ands should be tested in the next release to be declared stable.
   6,7 Fixed

[inikep]

Review of bfd0188

Per the review remarks, I did not treat missing certificate-signature verification or hostname verification as issues, because those behaviors are intentional in this library.

Findings

High

1. kmipclient/src/NetClientOpenSSL.cpp, kmipclient/include/kmipclient/NetClient.hpp, kmipclient/include/kmipclient/NetClientOpenSSL.hpp - timeout_ms does not apply to TCP connect / TLS handshake.

   The public API documents timeout_ms as a connect/read/write timeout, but the implementation only applies SO_RCVTIMEO / SO_SNDTIMEO after BIO_do_connect() succeeds. A stalled TCP connect or TLS handshake can therefore block until the operating system gives up, which breaks the advertised timeout contract and can hang pool growth or first-use connection establishment far longer than the configured limit.

2. kmipclient/src/IOUtils.cpp - IOUtils::send() assumes one transport write sends the whole KMIP request.

   NetClient::send() is defined as returning the number of bytes actually sent, but IOUtils::send() performs only one call and throws if sent < dlen. That makes the protocol layer reject valid short writes instead of retrying with the remaining bytes, which can turn a healthy connection into a failed request and leave the peer with a truncated KMIP message.

Medium

3. kmipcore/src/response_parser.cpp, kmipcore/include/kmipcore/kmip_basics.hpp - KMIP error codes are dropped when a server-side failure is raised as KmipException.

   ResponseParser::ensureSuccess() throws KmipException(formatOperationResult(item)), which uses the message-only constructor and leaves KmipException::code() at -1. The protocol response already contains Result Reason, so callers lose the machine-readable error code precisely on normal KMIP failure responses and can only parse the formatted text.

4. kmipcore/src/key_parser.cpp, kmipclient/src/KmipClient.cpp, kmipcore/include/kmipcore/secret.hpp - op_get_secret() returns a Secret whose typed state field is always KMIP_STATE_PRE_ACTIVE.

   KeyParser::parseGetSecretResponse() constructs Secret with a hard-coded KMIP_STATE_PRE_ACTIVE, and KmipClient::op_get_secret() later stores the real lifecycle state only in the generic attribute map. Code that reads secret.state gets stale / incorrect data even when the server returned a different state.

5. kmipcore/src/kmip_basics.cpp - Element::deserialize() uses signed left shifts when decoding wire bytes.

   Expressions such as (data[offset + 4] << 24) assemble TTLV fields after the uint8_t values have been promoted to int. For bytes with the top bit set, shifting into the sign bit is undefined behavior in C++, so malformed or edge-case TTLV data can drive the parser through UB instead of cleanly decoding or rejecting the message.

Low

6. kmipcore/src/kmip_formatter.cpp, kmipcore/include/kmipcore/kmip_basics.hpp - formatting a valid KMIP_TYPE_INTERVAL element throws instead of printing the value.

   The formatter handles KMIP_TYPE_INTERVAL by calling element->toInt(), but interval values are stored in the separate Interval variant, not the Integer variant. As a result, format_element() throws "Element is not Integer" for valid interval TTLV nodes.

[percona-ysorokin]

For exceptions with an error code it is better to use std::system_error

class KmipException  : public std::system_error {
public:
  KmipException (int native_error_code, const std::string &what);
};

const std::error_category &kmip_category() noexcept {
  class [[nodiscard]] category_impl : public std::error_category {
  public:
    [[nodiscard]] const char *name() const noexcept override {
      return "kmip";
    }
    [[nodiscard]] std::string message(int code) const override {
      // Convert Error code to std::string here
      return code_to_string(code);
    }
  };

  static const category_impl instance;
  return instance;
}

[[nodiscard]] inline std::error_code
make_kmip_error_code(int native_error_code) noexcept {
  return {native_error_code, kmip_category()};
}

KmipException::KmipException(int native_error_code, const std::string &what)
    : std::system_error{make_kmip_error_code(native_error_code), what} {}

[lukin-oleksiy]

Now KmipException is more complex and it handles protocol errors according to the KMIP 1.4 specification.
I think your suggestion was fullfiled.

[percona-ysorokin]

Here is a good example where including just xxx_fwd.hpp files would be enough.