Detect and disable ccache

[nettle]

We need to find the best way to disable ccache (which must be disabled in Bazel)
especially for the environments where it is not possible to remove (sudo apt-get remove).

Scope

1. Experiment and suggest ways to disable ccache
   • Note that CCACHE_DISABLE=1 does not work
2. Describe the best way to disable ccache
3. Detect ccache in the code (tools.bzl?)
4. Update documentation and code
   • Add description to ccache detection code (tools.bzl?)
   • README
   • CONTRIBUTING (if needed)

DoD (Definition of Done)

• Detection is added
• Documentation is updated

[furtib]

I am currently unable to reproduce this issue!
I am on Ubuntu 24.04, I have ccache installed, and following the documentation, I even "let ccache masquerade as the compiler" (I have done this to GCC, g++, clang and clang++)

[nettle]

I am currently unable to reproduce this issue! I am on Ubuntu 24.04, I have ccache installed, and following the documentation, I even "let ccache masquerade as the compiler" (I have done this to GCC, g++, clang and clang++)

Hm... that's strange.
Can you share the following:

which gcc
which g++
which clang

[furtib]

Ofc!

[efurtib|~]$ which gcc
/usr/local/bin/gcc
[efurtib|~]$ which g++
/usr/local/bin/g++
[efurtib|~]$ which clang
/usr/local/bin/clang
[efurtib|~]$ which clang++
/usr/local/bin/clang++

The original binaries are under /usr/bin

[furtib]

After a reboot, now I do have this error!!
Checking with our unit tests, it seems like it only affects the clang_* rules

[furtib]

Steps to reproduce:

• Install ccache
• shadow gcc/g++/clang/clang++ following this guide
• Rewrite the package_layout.json of codechecker, to point to the created symlinks

[furtib]

Just thinking out loud:
Adding the flag --action_env=CCACHE_DISABLE=1 fixes the problem.
Adding build --action_env=CCACHE_DISABLE=1 to the .bazelrc file of the project fixes the problem. (it adds this flag to all build jobs)
Adding export CCACHE_DISABLE=1 to the start of the script found in code_checker.bzl fixes the problem, for code_checker

[nettle]

Ofc!

[efurtib|]$ which gcc
/usr/local/bin/gcc
[efurtib|]$ which g++
/usr/local/bin/g++
[efurtib|]$ which clang
/usr/local/bin/clang
[efurtib|]$ which clang++
/usr/local/bin/clang++
The original binaries are under /usr/bin

This is quite unusual :) Maybe something new in ccache?
Typical (or old?) path to ccache is /usr/lib64/ccache/ or similar (at least I used see such kind of configuration in Ubuntu).
Can you also check ls -la /usr/local/bin/gcc please?

In RHEL I have the following:

which gcc
/usr/lib64/ccache/gcc

which g++
/usr/lib64/ccache/g++

which clang
/app/vbuild/RHEL8-x86_64/clang/16.0.3/bin/clang
# I use clang from module load

So, in my case clang rules are not affected (and CCACHE_DISABLE may work indeed),
however CodeChecker rules may fail because of the path with ccache:

python3 test/test.py
.........F...s....
======================================================================
FAIL: test_bazel_compile_commands (__main__.TestBasic)
Test: bazel build --build_tag_filters=compile_commands ...
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test.py", line 128, in test_bazel_compile_commands
  File "test/test.py", line 78, in grep_file
AssertionError: Could not find r'bin\/gcc' in '../bazel-bin/test/compile_commands_pass/compile_commands.json'

----------------------------------------------------------------------
Ran 18 tests in 41.438s

FAILED (failures=1, skipped=1)

Anyway, I think we have to take into account ccache path like /usr/lib64/ccache/gcc

[furtib]

Good idea!
I will open an issue for it!

[furtib]

I have come across a ton of anomalies in my setup (like clang not running because of missing files?!), so I wanted to recreate this bug in Docker, and I failed yet again. Do you guys have any idea why? The clang executable seems to be correctly overwritten by ccache. This is my Docker file:

FROM ubuntu:24.04

ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get update && \
    apt-get install -y \
    curl \
    wget \
    git \
    python3 \
    python3-pip \
    python3-venv \
    software-properties-common \
    unzip \
    lsb-release \
    gnupg \
    ca-certificates \
    g++-14 \
    clang-18 \
    clang-tools-18 \
    clang-tidy-18 \
    ccache && \
    rm -rf /var/lib/apt/lists/*

# Rename clang-18 to clang, etc.

RUN update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-18 100
RUN update-alternatives --install /usr/bin/clang-extdef-mapping \
    clang-extdef-mapping /usr/bin/clang-extdef-mapping-18 100
RUN update-alternatives --install /usr/bin/clang clang /usr/bin/clang-18 100
RUN update-alternatives --install /usr/bin/clang-tidy \
    clang-tidy /usr/bin/clang-tidy-18 100
RUN update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-14 100

# ccache shadow the compilers_
RUN ln -s /usr/bin/ccache /usr/local/bin/clang && \
    ln -s /usr/bin/ccache /usr/local/bin/clang++ && \
    ln -s /usr/bin/ccache /usr/local/bin/g++ && \
    ln -s /usr/bin/ccache /usr/local/bin/gcc

# Source: https://fbinfer.com/docs/getting-started
#RUN VERSION=1.1.0; \
#    curl -sSL "https://github.com/facebook/infer/releases/download/v$VERSION/infer-linux64-v$VERSION.tar.xz" \
#    | tar -C /opt -xJ && \
#    ln -s "/opt/infer-linux64-v$VERSION/bin/infer" /usr/local/bin/infer

# Setup python virtual enviroments
RUN python3 -m venv /venv
ENV VIRTUAL_ENV=/venv
ENV PATH=/venv/bin:$PATH

# Install CodeChecker and any other dependencies
RUN python3 -m pip install --upgrade pip \
    && python3 -m pip install --no-cache-dir codechecker

# Install bazelisk:
RUN VERSION=1.26.0; \
    wget "https://github.com/bazelbuild/bazelisk/releases/download/v$VERSION/bazelisk-linux-amd64" && \
    chmod +x bazelisk-linux-amd64 && \
    mv bazelisk-linux-amd64 /usr/local/bin/bazel && \
    USE_BAZEL_VERSION=6.5.0 bazel version

CMD echo $PATH && \
    which clang && \
    git clone https://github.com/furtib/codechecker_bazel.git && \
    cd codechecker_bazel/test && \
    git checkout ci_unit_test && \
    echo "6.5.0" > ../.bazelversion && \
    bazel version && \
    python3 test.py

[Szelethus]

For me, ccache symlinks by default, not sure what shenanigans I have in the background.

After installing ccache (sudo apt-get install ccache --no-install-recommends):

$ which g++
/usr/lib/ccache/g++
$ which clang
/usr/lib/ccache/clang

Some targets now fail, for instance, bazel build :test_fail:

ERROR: /path/to/codechecker_bazel/test/BUILD:59:11: Compiling test/src/ctu.cc failed: (Exit 1): gcc failed: error executing command (from target //test:test_ctu) /usr/lib/ccache/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++14' -MD -MF ... (remaining 17 arguments skipped)

After purging it (sudo apt-get purge ccache):

$ which g++
/usr/bin/g++
$ which clang
/usr/bin/clang

With regards to the proposed solution in #40: I'd prefer if whatever we come up with doesn't mess with the user's environment too much under the hood. I mean, bazel already does by creating a sandbox environment which can undo some mitigating steps to disable ccache, but still, we should first emit an error, where we also give offer an easy solution. Something like this:

ERROR: Running CodeChecker is not supported while ccache is enabled. Please disable it via --action_env=CCACHE_DISABLE=1

Or:

WARNING: Running CodeChecker is not supported while ccache is enabled. Ccache is disabled past this point.

[nettle]

Just thinking out loud: Adding the flag --action_env=CCACHE_DISABLE=1 fixes the problem. Adding build --action_env=CCACHE_DISABLE=1 to the .bazelrc file of the project fixes the problem. (it adds this flag to all build jobs) Adding export CCACHE_DISABLE=1 to the start of the script found in code_checker.bzl fixes the problem, for code_checker

To be honest I'm not a fan of adding extra options to Bazel commands (especially --action_env=...), and even to .bazelrc.
I would prefer to have some hardcore instructions on how to disable or even remove ccache from the environment :)
Besides we should not have ccache when we run Bazel anyway.
So, something like "sudo apt-get remove ccache" sounds fine to me, but... what if I cannot sudo?

[Szelethus]

To be honest I'm not a fan of adding extra options to Bazel commands (especially --action_env=...), and even to .bazelrc. I would prefer to have some hardcore instructions on how to disable or even remove ccache from the environment :) Besides we should not have ccache when we run Bazel anyway. So, something like "sudo apt-get remove ccache" sounds fine to me, but... what if I cannot sudo?

I think it's fine to give options. We could emit an error and direct the user to a documentation page, where we list some possible fixes:

• when someone so desires, use --action_env,
• remove ccache,
• change the build/CodeChecker/config/package_layout.json file in CodeChecker.

Mind that the last one will fix the analyzer binaries CodeChecker finds, but if bazel got hold of any other ccached analyzers/compilers along the way, we might still get an error (which is why #2 actually checked for it in two locations, one for the c++ compiler, one for the analyzers). Or, we could give a builtin solution similar to what #40 proposes.

WYD? I think I'd err on the side of giving an error and redirecting to a docs page.