Bug 4388 - Build system: parallel builds
Summary: Build system: parallel builds
Status: RESOLVED FIXED
Alias: None
Product: Samba 4.0
Classification: Unclassified
Component: Build (show other bugs)
Version: unspecified
Hardware: SGI IRIX
: P3 normal (vote)
Target Milestone: ---
Assignee: Jelmer Vernooij
QA Contact: samba4-qa@samba.org
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2007-02-09 18:30 UTC by g.fischer@ah-online.com
Modified: 2010-04-07 17:06 UTC (History)
0 users

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description g.fischer@ah-online.com 2007-02-09 18:30:55 UTC
irix 6.5.30, mipspro 7.4.x, gnu make 3.8.x, samba 400tp4.
-------------------------

using 'gmake -j [n]' is ignored. it's always only using one cpu at a time!
Comment 1 Andrew Bartlett 2007-02-09 21:45:51 UTC
I think we have parallel builds disabled, due to depenencey issues in the makefile, but I'll reassign to our build master and check...
Comment 2 g.fischer@ah-online.com 2007-02-09 22:29:46 UTC
(In reply to comment #1)
> I think we have parallel builds disabled, due to depenencey issues in the
> makefile, but I'll reassign to our build master and check...
> 

thanks, would highly appreciate this. especially since v4 is much bigger than 3.0.x.
the only rub i noticed in earlier version is that the initial headers like proto.h have to be made one after another or it dies. as soon as they're done all went fine using parallel.
Comment 3 Jelmer Vernooij 2007-02-10 06:50:29 UTC
Parallel builds are blocked by the fact that we don't have proper dependencies in the Makefile at the moment. The only way to have proper dependencies is autogenerating them. 

However, that plays silly games with our automatic header generation code as proper rules will trigger regenerating the prototype headers whenever one of the C files it is generated from changes (which, in turn will cause other C files to be regenerated), etc. For this reason, the original automatic dependencies code has been disabled.

At some point we had a hack that didn't write the new prototype header if it hadn't changed. However, that causes make to always think that particular header is out-of-date and attempt to regenerate it.

It very much looks like this bug can't be fixed properly without switching to some make alternative that doesn't rely on file change timestamps.
Comment 4 g.fischer@ah-online.com 2007-02-10 08:21:58 UTC
well no offence but that's a joke.

samba ain't one of these backyard apps and plays an important role in many infrastructures. looking at v4 i see gtk2 as a dep for swat2 - excuse me?
the compiled and stripped app has over 200mb without swat - what?
and to be a real pita jobserver is turned off, argh :-)

i don't know about your goals for v4 but i guess it's still supposed to be a server and not some 'ubuntu-gnome-toy' ...
Comment 5 Jelmer Vernooij 2007-02-10 09:21:21 UTC
the swat from samba4 doesn't in any way depend on gtk2. 

If you have suggestions on how to support the jobserver while not causing a complete rebuild of Samba when a single c file changes with a standard make, we're happy to look at it.
Comment 6 g.fischer@ah-online.com 2007-02-10 09:53:18 UTC
(In reply to comment #5)
> the swat from samba4 doesn't in any way depend on gtk2. 

iirc it's the config gui.
anyway a specific part does but that's not a big deal as long as it can be disabled via configure.

> 
> If you have suggestions on how to support the jobserver while not causing a
> complete rebuild of Samba when a single c file changes with a standard make,
> we're happy to look at it.
> 

i'd love to but since i'm not exactly a coder i'd have to spent a serious amount of time for that.
however how did you do it before v4? it always worked like a charm!
Comment 7 Jelmer Vernooij 2007-02-10 12:13:59 UTC
> (In reply to comment #5)
> > the swat from samba4 doesn't in any way depend on gtk2.  
> iirc it's the config gui.
> anyway a specific part does but that's not a big deal as long as it can be
> disabled via configure.
there are two different interfaces: one is the new swat (http-based, like it was previously), the other is the gtk+ stuff.

> > If you have suggestions on how to support the jobserver while not causing a
> > complete rebuild of Samba when a single c file changes with a standard make,
> > we're happy to look at it. 
> i'd love to but since i'm not exactly a coder i'd have to spent a serious
> amount of time for that.
> however how did you do it before v4? it always worked like a charm!
before v4 there wasn't that much autogenerated code. the dependencies are much more complicated now. 'make proto' won't just work because some headers require specific C files to be generated, which requires some compiler to be built, which needs prototypes, etc.

I guess we could do a two-stage approach where the first stage wouldn't support parallel builds. That would build things like compile_et, asn1_compile, etc. The second stage could then be built in parallel.
Comment 8 g.fischer@ah-online.com 2007-02-10 12:16:44 UTC
(In reply to comment #7)
> there are two different interfaces: one is the new swat (http-based, like it
> was previously), the other is the gtk+ stuff.

yes that's what i meant.

> I guess we could do a two-stage approach where the first stage wouldn't support
> parallel builds. That would build things like compile_et, asn1_compile, etc.
> The second stage could then be built in parallel.

sounds very good :-)
Comment 9 Jelmer Vernooij 2008-01-06 21:20:51 UTC
Maybe this will be fixed by the new buildsystem..
Comment 10 Matthias Dieter Wallnöfer 2008-06-07 12:56:37 UTC
How up-to-date is this issue?
Comment 11 Jelmer Vernooij 2008-06-07 20:39:36 UTC
(In reply to comment #10)
> How up-to-date is this issue?
Still valid, no news. 
Comment 12 Matthias Dieter Wallnöfer 2009-07-29 11:02:19 UTC
Update title (remove version number).
Comment 13 Matthias Dieter Wallnöfer 2010-02-13 07:27:23 UTC
Jelmer, does the new python build infrastructure change something?
Comment 14 Matthias Dieter Wallnöfer 2010-04-07 17:06:15 UTC
Update: the gmake build system doesn't and won't support this. But our new WAF-based one (just landed in the "master" branch - and will be available in the upcoming releases) finally supports parallel building. Please try it out (start it using "./autogen-waf.sh")!

This should be enough to mark this bug finally as "FIXED".