how to do some validation while reading (<< operator)?

[alanthie]

how to do some validation while reading (>> operator)?
can we read the first field and match with some kind of magical number?
or other method

[goblinhack]

In one of my games I write the size and version I expect when reading e.g.

in >> bits(my.t.version);
LOG("Read config: version = [%s]", my.t.version.c_str());
in >> bits(my.t.serialized_size);
LOG("Read config: serialized_size = %d", my.t.serialized_size);

if (my.t.serialized_size != sizeof(Config)) {
game_load_error = "bad save file header version";
return in;
}

any checking is really meant to be done by the upper layer, so adding version, size information is up to you - as it's a pretty lightweight tool, it doesn't do some of the versioning stuff that say protobufs might do - you roll your own on top - hth

[goblinhack]

e.g.

00:00:00.009: Read config: version = [0.0.1]
00:00:00.009: Read config: serialized_size = 632
00:00:00.009: Read config: window_pix_height = 1080
00:00:00.009: Read config: window_pix_width = 1728
00:00:00.009: Read config: config_pix_height = 1080
00:00:00.009: Read config: config_pix_width = 1728
00:00:00.009: Read config: debug_mode = 0
00:00:00.009: Read config: fps_counter = 0
00:00:00.009: Read config: gfx_allow_highdpi = 0
00:00:00.009: Read config: gfx_borderless = 1
00:00:00.009: Read config: gfx_fullscreen = 0
00:00:00.009: Read config: gfx_fullscreen_desktop = 0
00:00:00.009: Read config: gfx_vsync_enable = 1
00:00:00.009: Read config: mouse_wheel_lr_negated = 0
00:00:00.009: Read config: mouse_wheel_ud_negated = 0
00:00:00.009: Read config: music_volume = 42
00:00:00.009: Read config: sdl_delay = 10
00:00:00.009: Read config: sound_volume = 64

[alanthie]

Yep
Do you know that if you de-serialize an invalid file, you can crash your entire OS!
It happens to me
in >> bits(map) will read the 4 first bytes as the size of the map and can exhaust your memory if the number is big, crashing your system!

I resolved the issue by putting a signature string has the first field and reading the file first to validate the signature
The std::string is stored a 4 bytes length + the data
So, if the signature is ok, I then read with in >> bits(map)

Example:
const std::string SIGNATURE_RSAKEY = "RSAKEY_0001"; const long SIGNATURE_RSAKEY_POS = 4;

struct file_map_rsa
{
	std::string signature = cryptoAL::SIGNATURE_RSAKEY;
	std::map<std::string, cryptoAL::rsa::rsa_key> map;
	
    friend std::ostream& operator<<(std::ostream &out, Bits<file_map_rsa & > my)
    {
        out << bits(my.t.signature) 
			<< bits(my.t.map) ;
        return (out);
    }

    friend std::istream& operator>>(std::istream &in, Bits<file_map_rsa &> my)
    {
        in  >> bits(my.t.signature) 
			>> bits(my.t.map) ;
        return (in);
    }
};

bool read_map_rsa(const std::string& filenameDB, std::map<std::string, cryptoAL::rsa::rsa_key>& map_rsa, std::string& serr, bool check)
{
	if (file_util::fileexists(filenameDB) == false)
	{
		serr += "no file: " ; serr += filenameDB;
		return false;
	}
		
	if (check)
	{
		if (file_util::check_signature(filenameDB, cryptoAL::SIGNATURE_RSAKEY, cryptoAL::SIGNATURE_RSAKEY_POS, serr)  == false)
		{
			serr += "bad rsa file signature" ;
			return false;
		}
	}
	
	file_map_rsa m;
	
	std::ifstream infile;
	infile.open(filenameDB, std::ios_base::in);
	infile >> bits(m);
	infile.close();
	
	map_rsa = m.map;
	return true;
}

[[maybe_unused]] static bool check_signature(const std::string& filename, const std::string& SIGNATURE, long pos, std::string& err)
{
cryptoAL::cryptodata input_file;
if (input_file.read_from_file(filename) == false)
{
err += "ERROR reading file: "; err += filename;
return false;
}

    std::uint32_t sz_input = input_file.buffer.size();
    if (sz_input < SIGNATURE.size() + pos)
    {
        err += "ERROR invalid file size ";
        return false;
    }

	for(int i=0;i<SIGNATURE.size();i++)
	{
		if (input_file.buffer.getdata()[pos+i] != SIGNATURE[i])
		{
			 err += "ERROR invalid signature";
			 return false;
		 }
	}
	return true;
}

Usage:
std::string serr;
std::map<std::string, rsa::rsa_key> map_RSA;
bool ok = keymgr::read_map_rsa(filename, map_RSA, serr);

Sources:
https://codeberg.org/alainalain/encryptions

[goblinhack]

yeah you have to be careful with what is fed to it - I assume the write and read order was out of sync? i.e what was it reading as the map size? perhaps an exception handler wrapping this would help ? not sure how the header file could handle this - it could check for large map sizes.. but not ideal. Or it could add magic numbers so we know to expect a map - but that does bloat things a bit...

…

On Fri, 11 Apr 2025 at 14:37, Alain Lanthier ***@***.***> wrote: Yep Do you know that if you de-serialize an invalid file, you can crash your entire OS! It happens to me in >> bits(map) will read the 4 first bytes as the size of the map and can exhaust your memory if the number is big, crashing your system! I resolved the issue by putting a signature string has the first field and reading the file first to validate the signature The std::string is stored a 4 bytes length + the data So, if the signature is ok, I then read with in >> bits(map) Example: const std::string SIGNATURE_RSAKEY = "RSAKEY_0001"; const long SIGNATURE_RSAKEY_POS = 4; struct file_map_rsa { std::string signature = cryptoAL::SIGNATURE_RSAKEY; std::map<std::string, cryptoAL::rsa::rsa_key> map; friend std::ostream& operator<<(std::ostream &out, Bits<file_map_rsa & > my) { out << bits(my.t.signature) << bits(my.t.map) ; return (out); } friend std::istream& operator>>(std::istream &in, Bits<file_map_rsa &> my) { in >> bits(my.t.signature) >> bits(my.t.map) ; return (in); } }; bool read_map_rsa(const std::string& filenameDB, std::map<std::string, cryptoAL::rsa::rsa_key>& map_rsa, std::string& serr, bool check) { if (file_util::fileexists(filenameDB) == false) { serr += "no file: " ; serr += filenameDB; return false; } if (check) { if (file_util::check_signature(filenameDB, cryptoAL::SIGNATURE_RSAKEY, cryptoAL::SIGNATURE_RSAKEY_POS, serr) == false) { serr += "bad rsa file signature" ; return false; } } file_map_rsa m; std::ifstream infile; infile.open(filenameDB, std::ios_base::in); infile >> bits(m); infile.close(); map_rsa = m.map; return true; } [[maybe_unused]] static bool check_signature(const std::string& filename, const std::string& SIGNATURE, long pos, std::string& err) { cryptoAL::cryptodata input_file; if (input_file.read_from_file(filename) == false) { err += "ERROR reading file: "; err += filename; return false; } std::uint32_t sz_input = input_file.buffer.size(); if (sz_input < SIGNATURE.size() + pos) { err += "ERROR invalid file size "; return false; } for(int i=0;i<SIGNATURE.size();i++) { if (input_file.buffer.getdata()[pos+i] != SIGNATURE[i]) { err += "ERROR invalid signature"; return false; } } return true; } Usage: std::string serr; std::map<std::string, rsa::rsa_key> map_RSA; bool ok = keymgr::read_map_rsa(filename, map_RSA, serr); Sources: https://codeberg.org/alainalain/encryptions — Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADA3H7JWTPXMEIPGJ63LT32Y7ARJAVCNFSM6AAAAAB24IIQFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOJWHE2DCOBQGU> . You are receiving this because you commented.Message ID: ***@***.***> *alanthie* left a comment (goblinhack/c-plus-plus-serializer#4) <#4 (comment)> Yep Do you know that if you de-serialize an invalid file, you can crash your entire OS! It happens to me in >> bits(map) will read the 4 first bytes as the size of the map and can exhaust your memory if the number is big, crashing your system! I resolved the issue by putting a signature string has the first field and reading the file first to validate the signature The std::string is stored a 4 bytes length + the data So, if the signature is ok, I then read with in >> bits(map) Example: const std::string SIGNATURE_RSAKEY = "RSAKEY_0001"; const long SIGNATURE_RSAKEY_POS = 4; struct file_map_rsa { std::string signature = cryptoAL::SIGNATURE_RSAKEY; std::map<std::string, cryptoAL::rsa::rsa_key> map; friend std::ostream& operator<<(std::ostream &out, Bits<file_map_rsa & > my) { out << bits(my.t.signature) << bits(my.t.map) ; return (out); } friend std::istream& operator>>(std::istream &in, Bits<file_map_rsa &> my) { in >> bits(my.t.signature) >> bits(my.t.map) ; return (in); } }; bool read_map_rsa(const std::string& filenameDB, std::map<std::string, cryptoAL::rsa::rsa_key>& map_rsa, std::string& serr, bool check) { if (file_util::fileexists(filenameDB) == false) { serr += "no file: " ; serr += filenameDB; return false; } if (check) { if (file_util::check_signature(filenameDB, cryptoAL::SIGNATURE_RSAKEY, cryptoAL::SIGNATURE_RSAKEY_POS, serr) == false) { serr += "bad rsa file signature" ; return false; } } file_map_rsa m; std::ifstream infile; infile.open(filenameDB, std::ios_base::in); infile >> bits(m); infile.close(); map_rsa = m.map; return true; } [[maybe_unused]] static bool check_signature(const std::string& filename, const std::string& SIGNATURE, long pos, std::string& err) { cryptoAL::cryptodata input_file; if (input_file.read_from_file(filename) == false) { err += "ERROR reading file: "; err += filename; return false; } std::uint32_t sz_input = input_file.buffer.size(); if (sz_input < SIGNATURE.size() + pos) { err += "ERROR invalid file size "; return false; } for(int i=0;i<SIGNATURE.size();i++) { if (input_file.buffer.getdata()[pos+i] != SIGNATURE[i]) { err += "ERROR invalid signature"; return false; } } return true; } Usage: std::string serr; std::map<std::string, rsa::rsa_key> map_RSA; bool ok = keymgr::read_map_rsa(filename, map_RSA, serr); Sources: https://codeberg.org/alainalain/encryptions — Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADA3H7JWTPXMEIPGJ63LT32Y7ARJAVCNFSM6AAAAAB24IIQFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOOJWHE2DCOBQGU> . You are receiving this because you commented.Message ID: ***@***.***>

[alanthie]

yes you have to validate your file is some way before deserializing it blindly!
unless you want your OS to crash!