Bug 10124 - make test fails with SUITE: "basedir", TEST: "set CCACHE_BASEDIR" - Expected "cache miss" to be 1, got 0
Summary: make test fails with SUITE: "basedir", TEST: "set CCACHE_BASEDIR" - Expected ...
Status: CLOSED FIXED
Alias: None
Product: ccache
Classification: Unclassified
Component: ccache (show other bugs)
Version: dev
Hardware: All All
: P5 normal
Target Milestone: 3.2
Assignee: Joel Rosdahl
QA Contact: Joel Rosdahl
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-09-02 13:03 UTC by ishikawa
Modified: 2014-11-17 19:24 UTC (History)
0 users

See Also:


Attachments
current working directory needs to be stored in canonical form. (612 bytes, patch)
2013-09-05 09:36 UTC, ishikawa
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description ishikawa 2013-09-02 13:03:53 UTC
On a source tree that is stored in a directory that has a top level path component that is symlinked to 
another directory, make test fails with the following error.
 
SUITE: "basedir", TEST: "set CCACHE_BASEDIR" - Expected "cache miss" to be 1, got 0 

I think it has something do with the symlink in the path component since
the same test that is run on a different PC (64 bit Debian GNU/linux)
succeeded.

The following is a memo I prepared when I experienced
the issue on a 32-bit Debian.

--- 

The current ccache git source has a problem.

make test

fails with the following message.

SUITE: "basedir", TEST: "set CCACHE_BASEDIR" - Expected "cache miss" to be
1, got 0
cache directory
/home/ishikawa/repos/ccache/testdir.16250/.ccache
primary config
/home/ishikawa/repos/ccache/testdir.16250/.ccache/ccache.conf
secondary config      (readonly)    /usr/local/etc/ccache.conf
cache hit (direct)                     0
cache hit (preprocessed)               0
cache miss                             0
preprocessor error                     1
files in cache                         0
cache size                           0.0 kB
max cache size                       5.0 GB
TEST FAILED
Test data and log file have been left in testdir.16250
make: *** [test] Error 1

I investigated.
The error is due to the preprocessor error from the left ccache.log in the
temporary created test.NNNNN  directory.
>From the log:

[2013-09-02T21:37:52.836170 27715] Command line:
/home/ishikawa/repos/ccache/ccache gcc
-I/home/ishikawa/repos/ccache/testdir.16250/dir1/include -c src/test.c
[2013-09-02T21:37:52.836268 27715] Hostname: debian-vbox-ci
[2013-09-02T21:37:52.836297 27715] Working directory:
/home/ishikawa/repos/ccache/testdir.16250/dir1
[2013-09-02T21:37:52.836512 27715] Source file: src/test.c
[2013-09-02T21:37:52.836533 27715] Object file: test.o
[2013-09-02T21:37:52.836624 27715] Running preprocessor
[2013-09-02T21:37:52.836643 27715] Executing /usr/bin/gcc
-I../../../../../../new-hd1/extra/ishikawa/download/repos/ccache/testdir.16250/dir1/include
-E src/test.c
[2013-09-02T21:37:52.836661 27715] Unlink
/home/ishikawa/repos/ccache/testdir.16250/.ccache/tmp/test.tmp.debian-vbox-ci.27715.i
[2013-09-02T21:37:52.836702 27715] Unlink
/home/ishikawa/repos/ccache/testdir.16250/.ccache/tmp/tmp.cpp_stderr.debian-vbox-ci.27715
[2013-09-02T21:37:52.875022 27715] Unlink
/home/ishikawa/repos/ccache/testdir.16250/.ccache/tmp/test.tmp.debian-vbox-ci.27715.i
[2013-09-02T21:37:52.875144 27715] Unlink
/home/ishikawa/repos/ccache/testdir.16250/.ccache/tmp/tmp.cpp_stderr.debian-vbox-ci.27715
[2013-09-02T21:37:52.875186 27715] Preprocessor gave exit status 1
[2013-09-02T21:37:52.875215 27715] Failed; falling back to running the real
compiler
[2013-09-02T21:37:52.875233 27715] Executing /usr/bin/gcc
-I/home/ishikawa/repos/ccache/testdir.16250/dir1/include -c src/test.c
[2013-09-02T21:37:52.875270 27715] Acquired lock
/home/ishikawa/repos/ccache/testdir.16250/.ccache/c/stats.lock
[2013-09-02T21:37:52.875513 27715] Releasing lock
/home/ishikawa/repos/ccache/testdir.16250/.ccache/c/stats.lock
[2013-09-02T21:37:52.875534 27715] Unlink
/home/ishikawa/repos/ccache/testdir.16250/.ccache/c/stats.lock
[2013-09-02T21:37:52.875560 27715] Result: preprocessor error


So why did  preprocessor failed?
Executing /usr/bin/gcc
-I../../../../../../new-hd1/extra/ishikawa/download/repos/ccache/testdir.7413/dir1/include
-E src/test.c

I ran the compiler command manually. "cd" to testdir.16250/dir1/

/usr/bin/gcc -I/home/ishikawa/repos/ccache/testdir.16250/dir1/include
-c src/test.c

Funny, this worked manually!? Now why on earth, does the original
Command line: /home/ishikawa/repos/ccache/ccache gcc
-I/home/ishikawa/repos/ccache/testdir.7413/dir1/include -c src/test.c
gets transformed to the preprocessor command line:
Executing /usr/bin/gcc
-I../../../../../../new-hd1/extra/ishikawa/download/repos/ccache/testdir.7413/dir1/include
-E src/test.c
with relative pathname?

What is the content of path_stderr file when the cpp failed.

	if (status != 0) {
		if (!direct_i_file) {
			tmp_unlink(path_stdout);
		}
		tmp_unlink(path_stderr); <--- deleted after failure...
		cc_log("Preprocessor gave exit status %d", status);
		stats_update(STATS_PREPROCESSOR);
		failed();
	}

I patched the ccache.c so that this temporary file
is left behind after a preprocessor failure, and found
in another test run (note the different testdir.NNNN directory name),
the following file was left behind after the failure.

/home/ishikawa/repos/ccache/testdir.7413/.ccache/tmp/test.tmp.debian-vbox-ci.18985.i

It contained the following.
--- begin quote
src/test.c:1:18: fatal error: test.h: No such file or directory
compilation terminated.
---

So after invoking
/usr/bin/gcc
-I../../../../../../new-hd1/extra/ishikawa/download/repos/ccache/testdir.7413/dir1/include
-E src/test.c
Working directory: /home/ishikawa/repos/ccache/testdir.7413/dir1

it fails.

Again, why? On a hunch, I checked for the include directory.
it DOES NOT EXIST!!!
The preceding number of "../" to climb up the directory tree is WRONG!

>From "../dir1" directory, I needed the following path name to
properly enumerate the include files (i.e., test.h):

ls
../../../../../../../../new-hd1/extra/ishikawa/download/repos/ccache/testdir.7413/dir1/include
test.h

Note the required number of "../" is 8 while
somehow test.sh produced only six "../".

I am not sure if

- there is something wrong in the test.sh logic,

- there is something wrong in ccache source code which
  tries to rewrite the include path name into a relative one.

On my PC, /extra is symlink:
 ls -l /extra
lrwxrwxrwx 1 root root 14 Oct  3  2011 /extra -> /new-hd1/extra/

other components of the paths are ordinary directories.

---

TIA

PS: As I noted above, the test runs successfully with the
tree without symbolic name component on another computer, the
translation process that tries to base the pathname against the
basename of the ccache seems to buggy in the presence of symbolic link
in the pathname component.

TIA
Comment 1 Joel Rosdahl 2013-09-04 19:22:34 UTC
This looks like the same issue as bug #9801. Could you try if the attached patch works for you?
Comment 2 ishikawa 2013-09-05 07:12:17 UTC
(In reply to comment #1)
> This looks like the same issue as bug #9801. Could you try if the attached
> patch works for you?

Joel, thank you for your comment.

Things has gotten a little complicated.

With your patch ccache-3.1.9 source now runs its test fine:
make test finishes successfully (!)

But the current ccache git repository source still fails with the same error.

Then I realize that the current git repository has many more tests than the
current release: current release has

> PASSED: 176 assertions, 64 tests, 9 suites
                     **************************
whereas the current git repository has

> PASSED: 439 assertions, 86 tests, 10 suites
                     ***************************
so I assume that one of the new tests is failing in the current
git repository source.
Sorry, I should have mentioned this in my original post.

But your patch gives me a clue about where to look for potential problems.
I will sprinkle the source file with printf statements and see
what is transforming the header include directory with a series of "../", and a couple of them missing at that.

TIA
Comment 3 ishikawa 2013-09-05 09:36:52 UTC
Created attachment 9190 [details]
current working directory needs to be stored in canonical form.

I found the root cause.

The relative path generation routine assumes that both pathnames given to
it are in canonical forms (removing  unnecessary a/../b, or resolve symlinks).

In util.c:
/*
 * Compute a relative path from from (an absolute path to a directory) to to (a
 * path). Assumes that both from and to are well-formed and canonical. Caller
 * frees.
 */
char *
get_relative_path(const char *from, const char *to)


However, one important variable that contains a directory path is not
necessarily in canonical form.
That is, current_working_dir.

The attached patch fixes this issue.
(There are two places where current_working_dir is set after calling get_cwd().)
One of the places got it right (I think it is called after ccache reset,
but the initialization routine forgets to canonicalize current_working_dir.)

After the fix, the |make test| passes on the PC where
the source tree of ccache is stored in a directory which is symlinked to
another one: I always use the directory path that contains the symbolic link
in its path component for historical reasons and got bitten.

TIA

PS: Maybe get_relative_path (in a debug build for testing purposes) can
check that the passed pathnames are indeed in canonical form by calling
x_realpath(...) : which is expensive, and that is why only for debug build for testing purposes.
Comment 4 Joel Rosdahl 2013-10-17 19:32:18 UTC
(In reply to comment #3)
> I found the root cause.
> [...]
> However, one important variable that contains a directory path is not
> necessarily in canonical form. That is, current_working_dir.

It looks like I made a mistake when merging maint to master at some point - the setting of current_working_dir was intended to be done lazily in make_relative_path(), not in initialize(). I committed a fix and a test to master in c9f60ad112ad8e254880d78cd1e5d16da5c98c53.

Thank you very much for finding the root cause!
Comment 5 Joel Rosdahl 2014-11-17 19:24:25 UTC
Included in v3.2.