res_kmeans.py对不同的金融产品进行处理，出现sid和pid不是一一对应，一个sid可能对应多个pid，出现sid重复的现象，请问如何解决？

[mcstone1314]

【问题描述】
res_kmeans.py对不同的金融产品进行处理，出现sid和pid不是一一对应，一个sid可能对应多个pid，出现sid重复的现象
【问题举例】
举例：以下4个产品描述的sid均相同：<|sid_begin|><s_a_1661><s_b_472><s_c_7622><|sid_end|>

产品名称为日添利号，以科技创新和消费领域为主要投资方向，风格偏向平衡型，适合追求长期稳健收益的投资者。属于混合类 - 基金类产品，风险等级为R2。产品说明：本产品动态调整股债比例，捕捉市场机会，风险较低。历史收益水平(%)5.12。

产品名称为日添利精选期，主要投资于高收益类资产，投资风格偏稳健，适合风险承受能力较强的投资者。属于FOF - 基金类产品，风险等级为R4。产品说明：本产品属于FOF - 基金类产品，风险等级为R4。历史收益水平(%)7.49。

产品名称为日添利期，投资于稳健债市与流动性资产，风格偏稳健，适合保守型投资者。属于现金管理类 - 基金类产品，风险等级为R1。产品说明：流动性强，风险极低，适合短期闲置资金。历史收益水平(%)1.54。

产品名称为日添利期，聚焦于成长性企业投资，采取灵活配置策略，适合追求稳健增值的投资者。属于FOF - 基金类产品，风险等级为R4。产品说明：本产品属于FOF - 基金类产品，风险等级为R4。历史收益水平(%)6.36。
【诉求】

1. 对于OpenOneRec来说，为何将上述产品描述差异的产品生成相同sid？
2. 对于OpenOneRec来说，如何可以有效区分开sid，是产品描述中有效数字尽量不同？不同的描述放在总的产品描述前面还是后面？还是要满足多少字数以上才能区分开？
3. 对于相似的产品描述，是否只可能s_c不同，s_a和s_b一定相同？

[HaFred]

Interesting case here. My take is that the emb for these financial terms in a general corpus is quite similar, to make sid2pid into an injective mapping, tricks like collision detections may be worthwhile.

[mcstone1314]

Interesting case here. My take is that the emb for these financial terms in a general corpus is quite similar, to make sid2pid into an injective mapping, tricks like collision detections may be worthwhile.

Thanks for your advice. Here is my current implementation plan, and I would appreciate your feedback on whether it is reasonable:

1. Read the product text, call the Embedding API to obtain vectors, quantize them into integer codes using a pre-trained Residual K-Means Tokenizer, and generate the official-format SID.
2. In case of collisions (where different products produce the same SID), perturb the SID within the collision group based on PID and additional information.
   Only an offset is applied to the last layer (s_c), while the first two layers (s_a, s_b) remain unchanged to preserve semantic proximity.
   The offset method I am using is as follows:
   1）Map the PID and additional features (such as price, year) to a fixed-length offset vector using the mmh3 hash function.
   2）Repeat until no collisions remain or the maximum number of iterations is reached.