Add The Pertensor Quant And Fix some BUGs#458
Add The Pertensor Quant And Fix some BUGs#458Michael20070814 wants to merge 8 commits intoModelTC:mainfrom
Conversation
| type: model_type | ||
| path: model path | ||
| type: Qwen3 | ||
| path: /home/michael/Project/models/Qwen3-0.6B |
There was a problem hiding this comment.
do not use hardcode path and models, revert to the original content
| download: False | ||
| path: calib data path | ||
| path: /home/michael/Project/calib/pileval | ||
| n_sample: 128 |
There was a problem hiding this comment.
If not add the n_sample, here will emerge some problems.
[rank0]: AttributeError: 'EasyDict' object has no attribute 'n_sample'. Did you mean: 'n_samples'?
Just like this.
It seems that some spelling mistake had happened. Let me check it!
| seed: *seed | ||
| eval: | ||
| eval_pos: [transformed, fake_quant, fake_quant_wo_kv] #long_ppl eval not support pretrain eval pos | ||
| eval_pos: [] #long_ppl eval not support pretrain eval pos |
| act_static_cfg.update(self.config.calib.bs) | ||
| # KV cache 构造函数接收的是 num_samples / bsz， | ||
| # 这里把校准配置里的字段名映射成它实际需要的参数名。 | ||
| act_static_cfg['num_samples'] = self.config.calib.n_sample |
There was a problem hiding this comment.
use self.config.calib.n_samples
There was a problem hiding this comment.
That's the problem what causes that I must set the n_sample.
|
|
||
| # 按 LightLLM 的 kv_cache_calib.json 结构导出校准结果， | ||
| # 目前只支持它已经接入的 per_tensor / per_head 两种 KV 格式。 | ||
| def collect_calib_json(self): |
There was a problem hiding this comment.
The process of exporting calibration json file strictly relates with LightLLM. It's better to move this part to the utils folder.
There was a problem hiding this comment.
Yes. I will make it more moduled
llmc/__main__.py
Outdated
| # 动态分配模型 | ||
| model = MODEL_REGISTRY[config.model.type](config) | ||
|
|
||
| # 打印模型和tokenizer |
There was a problem hiding this comment.
remove this types of comment. And avoid using chinese.
llmc/__main__.py
Outdated
| eval_model(model, blockwise_opts, eval_list, eval_pos='transformed') | ||
| # 只有rank 0继续做保存和导出 | ||
| if int(os.environ['RANK']) == 0: | ||
| if 'save' in config and config.save.get('save_calib_json', False): |
There was a problem hiding this comment.
save_calib_json is not a clear config option. modify it after discussion.
There was a problem hiding this comment.
Renamed the config to make the export target explicit:
save_calib_json -> save_lightllm_kv_cache_calib
calib_json_name -> lightllm_kv_cache_calib_name
This export is specifically for LightLLM KV cache calibration, so the new naming is more precise.
2 participants
No description provided.