Bug 8460 - -fPIC detection in autoconf fails for clang
Summary: -fPIC detection in autoconf fails for clang
Status: CLOSED FIXED
Alias: None
Product: ccache
Classification: Unclassified
Component: ccache (show other bugs)
Version: 3.1.9
Hardware: x64 FreeBSD
: P5 normal
Target Milestone: 3.2
Assignee: Joel Rosdahl
QA Contact: Joel Rosdahl
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-09-16 11:48 UTC by c.kworr
Modified: 2015-03-07 15:54 UTC (History)
2 users (show)

See Also:


Attachments
compilation logs (6.62 KB, text/plain)
2011-09-26 09:29 UTC, c.kworr
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description c.kworr 2011-09-16 11:48:47 UTC
I really puzzled with this one and haven't been thinking that ccache is behind this.

When running configure script with ccache in path I get this:

checking for clang option to produce PIC... -fPIC
checking if clang PIC flag -fPIC works... no

configure:10712: checking for clang++ option to produce PIC
configure:10964: result: -fPIC
configure:10972: checking if clang++ PIC flag -fPIC works
configure:10990: clang++ -c -O2 -pipe -fno-strict-aliasing  -fPIC -DPIC conftest.cc >&5
clang++: warning: argument unused during compilation: '-D PIC'
configure:10994: $? = 0
configure:11005: result: no

This only happens on x64, on my other x86 machine this wouldn't happen.

Disabling ccache works.

I'll be happy to provide further logs, dumps or anything.
Comment 1 c.kworr 2011-09-16 14:01:35 UTC
After searching more I found that I can't build some packages with ccache or without it. Traces leading to libtool, as if I rebuild libtool without ccache error vanishes.

Rebuilding libtool without ccache doesn't fix the problem I described though.
Comment 2 Joel Rosdahl 2011-09-25 18:05:23 UTC
It looks like configure thinks that -fPIC doesn't work because the test produces stderr output ("warning: argument unused during compilation: '-D PIC'"). This is because clang warns for things that GCC doesn't warn about, in this case that "-D PIC" have no effect when compiling preprocessed code. See bug #8118.

ccache doesn't support clang yet, so it's in a sense just pure luck if it works at the moment. However, until clang is officially supported, a workaround should be to set CCACHE_CPP2 as this will avoid compiling preprocessed code.
Comment 3 c.kworr 2011-09-26 06:04:21 UTC
Well, it's not that simple. The problem is not in -fPIC detection code.

Notice the command that autoconf tries to execute. The same command completes successfully on the same source file. But it seems that ccache modifies command before running it. clang is always noisy and strict and it says there's another '-D PIC' in the command, but command contains only '-DPIC' without space. It seems that ccache does here some gcc magic which clang doesn't understand.

I'll try to retest with CCACHE_CPP2 though.
Comment 4 Joel Rosdahl 2011-09-26 06:14:13 UTC
Please also set CCACHE_LOGFILE to get a debug log. That way, you will get more information on what ccache does, which commmands it runs, etc.
Comment 5 c.kworr 2011-09-26 09:29:11 UTC
Created attachment 6936 [details]
compilation logs

I've tried again autoconfiguration for -fPIC. I've also added CCACHE_RECACHE to prevent cache lookup.

In both cases autoconf sees this:

configure:11180: checking for clang++ option to produce PIC
configure:11432: result: -fPIC
configure:11440: checking if clang++ PIC flag -fPIC works
configure:11458: clang++ -c -O2 -pipe -I/usr/local/include -fno-strict-aliasing -I/usr/local/include -fPIC -DPIC conftest.cc >&5
clang++: warning: argument unused during compilation: '-I /usr/local/include'
clang++: warning: argument unused during compilation: '-I /usr/local/include'
clang++: warning: argument unused during compilation: '-D PIC'
configure:11462: $? = 0
configure:11473: result: no

I'm totally puzzled how -I option was specified more then one time (or even triple?).
Comment 6 c.kworr 2011-09-26 09:30:50 UTC
Tried this script:

#!/bin/sh -

echo $* >> /tmp/full.log
ccache /usr/bin/clang $*

Nothing intresting except now autoconf result is positive.
Comment 7 c.kworr 2011-09-26 09:33:50 UTC
(In reply to comment #6)
> Tried this script:
> 
> #!/bin/sh -
> 
> echo $* >> /tmp/full.log
> ccache /usr/bin/clang $*
> 
> Nothing intresting except now autoconf result is positive.

Positive but only for clang++...
Comment 8 c.kworr 2011-09-26 09:46:00 UTC
(In reply to comment #7)
> > Nothing intresting except now autoconf result is positive.
> Positive but only for clang++...

Sorry, that was only because of incorrect link to ccache.
Comment 9 Joel Rosdahl 2011-09-26 18:15:01 UTC
(In reply to comment #5)
> In both cases autoconf sees this: [...]

It doesn't make sense that clang++ emits those warnings when you use CCACHE_CPP2. As can be seen from the ccache log, ccache runs this commands (output name tweaked):

  /usr/bin/clang++ -c -O2 -pipe -I/usr/local/include -fno-strict-aliasing -I/usr/local/include -fPIC -DPIC -E conftest.cc
  /usr/bin/clang++ -c -O2 -pipe -I/usr/local/include -fno-strict-aliasing -I/usr/local/include -fPIC -DPIC -o file.o.foo conftest.cc

If you run those commands manually, does clang++ emit the same warnings?

> I'm totally puzzled how -I option was specified more then one time (or even
> triple?).

What's strange about that?
Comment 10 c.kworr 2011-09-27 15:11:09 UTC
(In reply to comment #9)
> (In reply to comment #5)
> > In both cases autoconf sees this: [...]
> 
> It doesn't make sense that clang++ emits those warnings when you use
> CCACHE_CPP2. As can be seen from the ccache log, ccache runs this commands
> (output name tweaked):
> 
>   /usr/bin/clang++ -c -O2 -pipe -I/usr/local/include -fno-strict-aliasing
> -I/usr/local/include -fPIC -DPIC -E conftest.cc
>   /usr/bin/clang++ -c -O2 -pipe -I/usr/local/include -fno-strict-aliasing
> -I/usr/local/include -fPIC -DPIC -o file.o.foo conftest.cc
> 
> If you run those commands manually, does clang++ emit the same warnings?

I'll include full list of my tests, just in case. Mentioned file is:

int some_variable = 0;

Trying to run first command without ccache:

[] /usr/bin/clang++ -c -O2 -pipe -I/usr/local/include -fno-strict-aliasing -fPIC -DPIC -E conftest.cc
# 1 "conftest.cc"
# 1 "conftest.cc" 1
# 1 "<built-in>" 1
# 1 "<built-in>" 3
# 137 "<built-in>" 3
# 1 "<command line>" 1
# 1 "<built-in>" 2
# 1 "conftest.cc" 2
int some_variable = 0;

Same with ccache:

[] clang++ -c -O2 -pipe -I/usr/local/include -fno-strict-aliasing -fPIC -DPIC -E conftest.cc
# 1 "conftest.cc"
# 1 "conftest.cc" 1
# 1 "<built-in>" 1
# 1 "<built-in>" 3
# 137 "<built-in>" 3
# 1 "<command line>" 1
# 1 "<built-in>" 2
# 1 "conftest.cc" 2
int some_variable = 0;

Second command without ccache:

[] /usr/bin/clang++ -c -O2 -pipe -I/usr/local/include -fno-strict-aliasing -I/usr/local/include -fPIC -DPIC conftest.cc

Same with ccache:

[] clang++ -c -O2 -pipe -I/usr/local/include -fno-strict-aliasing -I/usr/local/include -fPIC -DPIC conftest.cc
clang++: warning: argument unused during compilation: '-I /usr/local/include'
clang++: warning: argument unused during compilation: '-I /usr/local/include'
clang++: warning: argument unused during compilation: '-D PIC' 

> > I'm totally puzzled how -I option was specified more then one time (or even
> > triple?).
> 
> What's strange about that?

I dunno.

Running second command with ccache but only one -I parameter results in:

[] clang++ -c -O2 -pipe -I/usr/local/include -fno-strict-aliasing -fPIC -DPIC conftest.cc

And no errors...

But I think I found my poison. Here is a piece of code:

#include <stdio.h>
#include <unistd.h>

int main(int argc, char** argv) {
        const char* cmd = "/usr/bin/clang++";

        for(int i = 0; i < argc; i++) {
                printf("%s ", argv[i]);
        }

        printf("\n");

        execvp(cmd, argv);
};

Nothing hard to explain - prints command line and executes another binary.

And here is the sample output:

[] ./a.out -c -O2 -pipe -I/usr/local/include -fno-strict-aliasing -I/usr/local/include -fPIC -DPIC conftest.cc
./a.out -c -O2 -pipe -I/usr/local/include -fno-strict-aliasing -I/usr/local/include -fPIC -DPIC conftest.cc
/home/arcade/a.out -cc1 -triple x86_64-unknown-freebsd9.0 -emit-obj -disable-free -main-file-name conftest.cc -pic-level 2 -mdisable-fp-elim -relaxed-aliasing -masm-verbose -mconstructor-aliases -munwind-tables -target-cpu x86-64 -momit-leaf-frame-pointer -coverage-file conftest.o -resource-dir /home/arcade/../lib/clang/3.0 -D PIC -I /usr/local/include -I /usr/local/include -O2 -fdeprecated-macro -ferror-limit 19 -fmessage-length 142 -fcxx-exceptions -fexceptions -fdiagnostics-show-option -fcolor-diagnostics -o conftest.o -x c++ conftest.cc

So there is another call for the binary and now I perfectly see the changes made to the command line: '-DPIC' => '-D PIC'. And this second call is complaining about something.

I'll try to retest all of this with 'fork'.
Comment 11 Mike Frysinger 2012-07-06 04:34:52 UTC
clang should not be complaining about unused -I/-D flags when compiling (-c).  if it is, that's a bug to report to the clang authors.
Comment 12 c.kworr 2012-07-06 07:15:29 UTC
This one is not about how clang complains but how clang executes.

I rerun my test on current version of clang/ccache and they keep giving me the same result:

No ccache:

arcade@green\~/work/test> clang -c -O2 -pipe -march=core2 -I/usr/local/include -fno-strict-aliasing -I/usr/local/include -fno-rtti -fno-exceptions test1.c
./clang -c -O2 -pipe -march=core2 -I/usr/local/include -fno-strict-aliasing -I/usr/local/include -fno-rtti -fno-exceptions test1.c 
/home/arcade/work/test/clang -cc1 -triple x86_64-unknown-freebsd9.0 -emit-obj -disable-free -main-file-name test1.c -mrelocation-model static -mdisable-fp-elim -relaxed-aliasing -masm-verbose -mconstructor-aliases -munwind-tables -target-cpu core2 -momit-leaf-frame-pointer -coverage-file test1.o -resource-dir /home/arcade/work/test/../lib/clang/3.1 -I /usr/local/include -I /usr/local/include -fmodule-cache-path /var/tmp/clang-module-cache -O2 -fdebug-compilation-dir /home/arcade/work/test -ferror-limit 19 -fmessage-length 186 -mstackrealign -fno-rtti -fgnu-runtime -fobjc-runtime-has-arc -fobjc-runtime-has-weak -fobjc-dispatch-method=non-legacy -fdiagnostics-show-option -fcolor-diagnostics -o test1.o -x c test1.c 

Ccache:

arcade@green\~/work/test> clang -c -O2 -pipe -march=core2 -I/usr/local/include -fno-strict-aliasing -I/usr/local/include -fno-rtti -fno-exceptions test1.c
./clang -c -O2 -pipe -march=core2 -I/usr/local/include -fno-strict-aliasing -I/usr/local/include -fno-rtti -fno-exceptions test1.c

The problem may also be in file naming differences. GCC consists of many different executables and clang is the only one. So when clang re-executes itself with it's own parameters those parameters are filtered through ccache too.

Anyway I'll try to bug clang developers.
Comment 13 Mike Frysinger 2012-07-06 14:55:13 UTC
from ccache's perspective, the internal behavior of clang doesn't matter.  if clang is incorrectly warning about -D/-I flags when it is compiling something, then it's a bug to take up with the clang developers.  ccache should not be working around broken behavior in clang.
Comment 14 Joel Rosdahl 2012-07-19 21:19:56 UTC
(In reply to comment #12)
> This one is not about how clang complains but how clang executes.

I'm still not sure about that. I haven't seen anything that makes me suspect anything other than that the problem is that clang complains about -D/-I and friends when run with -E.

(The fact that clang calls itself a second time shouldn't matter: clang can't see that it's masqueraded by ccache since ccache sets up argv[0] (passed to execvp when executing the compiler) to be the compiler, unlike your test program which has itself in argv[0].)

Please try this script:

===[cut here]=========================================================
#!/bin/sh

export CCACHE_DIR=ccachedir.$$
command="/usr/bin/clang -I/foo -c ccachetest-$$.c"

run() {
    rm -rf $CCACHE_DIR
    echo "int variable = 0;" >ccachetest-$$.c
    echo "--- $*"
    $*
    rm -rf $CCACHE_DIR ccachetest-$$.c
}

run $command
run ccache $command
run env CCACHE_CPP2=1 ccache $command
===[cut here]=========================================================

When run with clang version "3.0-6ubuntu3", it gives me this output:

===[cut here]=========================================================
--- /usr/bin/clang -I/foo -c ccachetest-7495.c
--- ccache /usr/bin/clang -I/foo -c ccachetest-7495.c
clang: warning: argument unused during compilation: '-I /foo'
--- env CCACHE_CPP2=1 ccache /usr/bin/clang -I/foo -c ccachetest-7495.c
===[cut here]=========================================================

That is, when ccache doesn't run clang with -E (the CCACHE_CPP2=1 case), clang doesn't complain. The tricky thing here is that once you've built without CCACHE_CPP2=1, then the "argument unused during compilation" warnings have also been stored in the cache, and then it doesn't matter if you later run with CCACHE_CPP=1 since ccache will give you the previous result from the cache. That's why my script above makes sure to run with a separate clean ccache directory each time.

> [...] So when clang re-executes itself with it's own parameters those
> parameters are filtered through ccache too.

If this were the case, you would see the re-execution in ccache's log, which I guess you haven't?
Comment 15 Joel Rosdahl 2012-07-19 21:27:51 UTC
(In reply to comment #13)
> ccache should not be working around broken behavior in clang.

I'm not against adding special cases for clang or other compilers as long as it doesn't affect other compilers negatively. ccache already has compiler-specific workarounds: there is code for handling several GCC bugs, there's special treatment for a broken distcc version shipped by Apple, etc.
Comment 16 c.kworr 2012-07-20 09:09:24 UTC
Yes CCACHE_CPP2 works for me and behaves correctly. I think I was totally wrong about the cause.

The root of the problem is that clang correctly points out that none of preprocessing flags affects compilation result. GCC just skips them.

There are other opinions on fixing this.

http://llvm.org/bugs/show_bug.cgi?id=13282

How about stripping preprocessing flags when running over preprocessed source? As Richard Smith says this perfectly works for gcc too and this seems quite logical for me.
Comment 17 Joel Rosdahl 2012-07-20 14:53:49 UTC
(In reply to comment #16)
> How about stripping preprocessing flags when running over preprocessed source?

Sure (as commented in http://www.mail-archive.com/ccache@lists.samba.org/msg00823.html, but you're perhaps not reading the ccache list?). One problem is finding a good list of options to strip. Suggestions are welcome.

> As Richard Smith says this perfectly works for gcc too and this seems quite
> logical for me.

There are other compilers than GCC and clang (and many older GCC versions) to consider as well, but I think it's likely that there will be no problem.
Comment 18 Joel Rosdahl 2013-01-05 17:28:08 UTC
Various clang-related improvements have been made on the master branch.

Volodymyr, could you try if the current ccache master branch works for you?
Comment 19 Bryan Drewery 2013-04-23 10:50:08 UTC
(In reply to comment #18)
> Various clang-related improvements have been made on the master branch.
> 
> Volodymyr, could you try if the current ccache master branch works for you?

I've tried with master on d1b2219297 and this is still an issue with clang. Defining CCACHE_CCP2=1 is still required.

Example call that works without ccache or with CCACHE_CPP2=1:

/usr/local/libexec/ccache/world/cc  -O2 -pipe -march=pentium4 -I/usr/src/lib/libelf -I. -DLIBELF_TEST_HOOKS -std=gnu99 -Qunused-arguments -fstack-protector -Wsystem-headers -Werror -Wall -Wno-format-y2k -W -Wno-unused-parameter -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Wreturn-type -Wcast-qual -Wwrite-strings -Wswitch -Wshadow -Wunused-parameter -Wcast-align -Wchar-subscripts -Winline -Wnested-externs -Wredundant-decls -Wold-style-definition -Wmissing-variable-declarations -Wno-pointer-sign -Wno-empty-body -Wno-string-plus-int -c /usr/src/lib/libelf/elf_scn.c -o elf_scn.o
/usr/src/lib/libelf/elf_scn.c:198:41: error: equality comparison with extraneous parentheses [-Werror,-Wparentheses-equality]
 if (((&e->e_u.e_elf.e_scn)->stqh_first == ((void *)0))) {
      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~
/usr/src/lib/libelf/elf_scn.c:198:41: note: remove extraneous parentheses around the comparison to silence this warning
 if (((&e->e_u.e_elf.e_scn)->stqh_first == ((void *)0))) {
     ~                                  ^             ~
/usr/src/lib/libelf/elf_scn.c:198:41: note: use '=' to turn this equality comparison into an assignment
 if (((&e->e_u.e_elf.e_scn)->stqh_first == ((void *)0))) {
                                        ^~
                                        =
1 error generated.


Line of code:

    if (STAILQ_EMPTY(&e->e_u.e_elf.e_scn)) {

The macro:

    #define STAILQ_EMPTY(head)      ((head)->stqh_first == NULL)
Comment 20 c.kworr 2013-07-10 12:49:16 UTC
@Bryan: I'd say that what you do describe with "#define" is totally different bug that would take place with any other compiler. It's the result of:

    #define MACRO(x) (x == NULL)

    if (MACRO(y)) {

Yielding this after preprocessing:

    if (MACRO((y == NULL)) {
    ^--------^           ^-^ - original code
              ^---------^    - macro substitute

Thus this would give us extraneous parentheses.

This problem can be mitigated by not doing full prepossession pass but reverting to unwind only #include's.

However I can't reproduce this bug with this sample:

    #define MACRO(x) ((int)x == (2 - 1))

    int main(void) {
      if (MACRO(1)) {
        return(0);
      } else {
        return(1);
      }
    }
    
This file perfectly compiles with ccache and without CCACHE_CPP2. My current compiler is clang3.3.
Comment 21 Luboš Luňák 2013-11-29 08:42:13 UTC
Isn't this simply a duplicate of bug #8118?
Comment 22 Joel Rosdahl 2014-08-09 09:38:24 UTC
> Isn't this simply a duplicate of bug #8118?

Since Bryan Drewery tested d1b22192 which contains 1ea5033e (fix of #8118), I guess not?

Has anybody been able to provide a reduced test case that demonstrates the problem?
Comment 23 Melvyn Sopacua 2014-08-16 06:39:43 UTC
(In reply to comment #22)

> Has anybody been able to provide a reduced test case that demonstrates the
> problem?

A variant of this problem is logged here:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=192158

I've tried to reduce it to a testcase, but so far that didn't work. CPPFLAGS is definitely the trigger, removing it also solved other ports suffering from this problem.
Comment 24 Joel Rosdahl 2015-02-07 15:14:08 UTC
Closing this now since I get believe that the originally reported bug has been resolved.