branch-4.1:[enhancement](filecache) filecache meta persist PRs compilation#61581
Open
freemandealer wants to merge 21 commits intoapache:branch-4.1from
Open
branch-4.1:[enhancement](filecache) filecache meta persist PRs compilation#61581freemandealer wants to merge 21 commits intoapache:branch-4.1from
freemandealer wants to merge 21 commits intoapache:branch-4.1from
Conversation
Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
add check for external data (lakehouse data) when get_tablet_id to avoid out-of-range exception Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
Removed duplicate definition of file_cache_keep_schema_change_output_min_hit_ratio.
Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
Signed-off-by: zhengyu <zhangzhengyu@selectdb.com>
…o task-branch-4-1-pick-persist-refactor # Conflicts: # be/src/cloud/cloud_tablet.cpp # be/src/exec/schema_scanner.cpp # be/src/io/cache/block_file_cache.cpp # be/src/io/cache/file_block.cpp # be/src/io/cache/fs_file_cache_storage.cpp # be/src/io/cache/fs_file_cache_storage.h # be/src/olap/rowset/beta_rowset_reader.cpp # gensrc/thrift/Descriptors.thrift
add changing type to Normal when TTL type & TTL set to 0 to make ttl queueu size correct
1. Drop missing tablet ids on NOT_FOUND to stop repeated TTL meta lookups and log spam 2. add bvar for tablet-id set size to monitor cleanup behavior
cache directory could be inconsistence with filecache meta store somehow, so extend FSFileCacheStorage with inode-based stats, leak-cleaner thread, and orphan cleanup helpers to clean such leakage.
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
16 tasks
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pick the following PRs/Commits:
[enhancement](filecache) use rocksdb to persist cache block meta #57072 -> 30406d6
[enhancement](filecache) fine-grained cache space observation #57783 -> 4cbae8a
[refactor](filecache) ttl management refactoring #57922 -> 0918952
[fix](filecache) fix out-of-range exception when external query #58082 -> 9f9a3e4
[fix](filecache) fix observation bug in multi-BE cluster #59259 -> 9f2c6ed
[enhancement](filecache) add filesystem leak cleaner #59269 -> e010d8d
[fix](filecache) load meta exit with file NOT_FOUND exception #59311 -> d336eed
[fix](filecache) reset_range dose not update shadow queue causing large cache size #59314 -> f7aa74e
[fix](filecache) add OFFSET column for table file_cache_info #59645 -> c4c3d14
[fix](filecache) correct ttl metrics #60252 -> b1585e1
[fix](filecache) add ttl mgr NOT_FOUND cleanup #60269 -> 26f44b5
[fix](filecache) add OFFSET column for table file_cache_info #59645 -> c4c3d14
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)