Bug 8063 - cannot upgrade from a11 to a14
Summary: cannot upgrade from a11 to a14
Alias: None
Product: Samba 4.0
Classification: Unclassified
Component: Tools (show other bugs)
Version: unspecified
Hardware: x86 Linux
: P5 normal (vote)
Target Milestone: ---
Assignee: Matthieu Patou
QA Contact: samba4-qa@samba.org
Depends on:
Reported: 2011-04-06 11:54 UTC by Dirk Pauli
Modified: 2011-06-20 13:01 UTC (History)
0 users

See Also:

upgrade script output (1.28 KB, text/plain)
2011-05-01 16:34 UTC, Dirk Pauli
no flags Details
Example of log of upgradeprovision (24.84 KB, application/octet-stream)
2011-05-31 08:12 UTC, Matthieu Patou
no flags Details
screenshots of upgrade process (414.29 KB, application/pdf)
2011-06-02 14:53 UTC, Dirk Pauli
no flags Details
upgradescript output (1.29 KB, text/plain)
2011-06-04 19:55 UTC, Dirk Pauli
no flags Details
new output (1.32 KB, text/plain)
2011-06-04 20:44 UTC, Dirk Pauli
no flags Details
ldbsearch output (406.67 KB, text/plain)
2011-06-05 12:37 UTC, Dirk Pauli
no flags Details
First patch, fix the origin of refsddl (1.15 KB, patch)
2011-06-05 14:33 UTC, Matthieu Patou
no flags Details
Second patch, accept more variety of SD (3.63 KB, patch)
2011-06-05 14:33 UTC, Matthieu Patou
no flags Details
debug output after applying patches (1.24 MB, text/plain)
2011-06-10 21:30 UTC, Dirk Pauli
no flags Details
patch for last problem (1.08 KB, patch)
2011-06-11 13:00 UTC, Matthieu Patou
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Dirk Pauli 2011-04-06 11:54:37 UTC
When trying to run upgradeprovision after building the new version, this script is always running into an error:
- upgradeprovision without parameters is telling me to use --full as the dynamic dns settings in the old version omit "quick" upgrade
- upgradeprovision --full show the following error output:
Creating a reference provision
Copy previlege
Update base samdb by searching difference with reference one
Starting update of samdb
There are 1 missing objects
Reloading a merged schema, which might trigger reindexing so please be patient
Schema reloaded!
Exception during upgrade of samdb:
Traceback (most recent call last):
  File "./source4/scripting/bin/upgradeprovision",line 1099, in update_partition  
    provisionUSNs, names.invocation)
  File "./source4/scripting/bin/upgradeprovision", line 908, in update_present
    if get_diff_sddls(refsddl, cursddl) == "":
  File "bin/python/samba/upgradehelpers.py" line 487, in get_diff_sddls
    if hash_new["owner"] != hash_ref["owner"]:
KeyError: 'owner'
Update failed

Using ubuntu 10.10, old samba version is Version 4.0.0alpha12-GIT-dace013, new samba version is Version 4.0.0alpha15-GIT-6d0be9e
Comment 1 Matthias Dieter Wallnöfer 2011-04-06 12:53:28 UTC
For ekacnet.
Comment 2 Dirk Pauli 2011-04-24 10:17:05 UTC
any news on how to fix this already?
Comment 3 Matthieu Patou 2011-04-24 10:42:43 UTC
Thanks for the headup I didn't had a look yet on this, but I tried yesterday for another problem to upgrade from alpha11 to alpha15 and it was ok.

Would you mind trying with the new alpha15 release and also use the --debugall flag on command line for more informations ?

Comment 4 Dirk Pauli 2011-05-01 16:34:20 UTC
Created attachment 6436 [details]
upgrade script output

Hi Matthieu,
unfortunately, it still do not work. I retried with Version 4.0.0alpha15-GIT-d874279. The error message is still the same.
Please find attached the script output with --debugall flag as requested.
Please also note that I needed to start the script from the samba directory: ./source4/scripting/bin/upgradeprovision
If I tried it to start from the source4 dir as stated in the upgradeing-samba4.txt, I get always the error "Import Error: no module named ldb".

Comment 5 Matthieu Patou 2011-05-01 17:20:07 UTC
What about of the --debugall flag ? can you use it and post here the output ? it will gives me some clues !
Comment 6 Dirk Pauli 2011-05-02 09:23:44 UTC
the file I attached yesterday (upgrade script output) was already generated with the --debugall flag of the script.

Let me know if you need any further information
Comment 7 Dirk Pauli 2011-05-30 20:01:52 UTC
were you able to find out anything on this in the meantime?

Comment 8 Matthieu Patou 2011-05-31 08:12:07 UTC
Well the output is strange.

Are you sure that you run upgradeprovision --full --debugall ? 
See my log for --debugchange and you'll see that it's much important than yours already.

Comment 9 Matthieu Patou 2011-05-31 08:12:42 UTC
Created attachment 6501 [details]
Example of log of upgradeprovision
Comment 10 Dirk Pauli 2011-06-02 14:53:52 UTC
Created attachment 6514 [details]
screenshots of upgrade process
Comment 11 Dirk Pauli 2011-06-02 14:56:09 UTC
Hi Matthieu,
I am pretty sure I did. I just retried after compiling the latest available source - unfortunately with the same result as earlier tries. See screenshots of what I did attached.

Comment 12 Matthieu Patou 2011-06-02 15:10:51 UTC
Ok I trust you,

can you try to apply this patch:

diff --git a/source4/scripting/python/samba/upgradehelpers.py b/source4/scripting/python/samba/upgradehelpers.py
index 16e4ea0..185d997 100755
--- a/source4/scripting/python/samba/upgradehelpers.py
+++ b/source4/scripting/python/samba/upgradehelpers.py
@@ -372,6 +372,8 @@ def get_diff_sddls(refsddl, cursddl):
     txt = ""
+    print cursddl
+    print refsddl
     hash_new = chunck_sddl(cursddl)
     hash_ref = chunck_sddl(refsddl)

and rerun and post the output
Comment 13 Dirk Pauli 2011-06-04 19:55:05 UTC
Created attachment 6517 [details]
upgradescript output

attached the file with the script output after applying the patch. Hope this helps narrowing down the problem
Comment 14 Matthieu Patou 2011-06-04 20:05:05 UTC
Ok that's clearer, I'm still a bit surprised that's why I'd like to know which object has this DN, can you add this patch as well ? and post the output ? 

diff --git a/source4/scripting/bin/upgradeprovision b/source4/scripting/bin/upgradeprovision
index e58a264..86bf818 100755
--- a/source4/scripting/bin/upgradeprovision
+++ b/source4/scripting/bin/upgradeprovision
@@ -907,6 +907,7 @@ def update_present(ref_samdb, samdb, basedn, listPresent, usns, invocationid):
                     refsddl = cursd.as_sddl(names.domainsid)
+                    print current[0]["dn"]
                     if get_diff_sddls(refsddl, cursddl) == "":
                        message(CHANGE, "sd are identical")
Comment 15 Matthieu Patou 2011-06-04 20:06:01 UTC
Also we can try get in touch on IRC it would ease the back and forth dialog.
Comment 16 Dirk Pauli 2011-06-04 20:44:08 UTC
Created attachment 6518 [details]
new output

new output after 2nd patch . P.S. I am currently on on #samba-technical
Comment 17 Matthieu Patou 2011-06-04 21:47:45 UTC
Well you should have something like this:


That's strange that the reference provision have also a broken Security descriptor.

Could you try to create a test provision like this:

./source4/setup/provision --realm paulis-home.local --domain PAULIS-HOME --server-role=dc --targetdir=/tmp/tests

and do then ./bin/ldbsearch -H /tmp/tests/private/sam.ldb ntsecuritydescriptor and post the output ? 

By doing so I want to see if a standalone provision is able to create the correct security descriptor.
Comment 18 Dirk Pauli 2011-06-05 12:37:54 UTC
Created attachment 6521 [details]
ldbsearch output

as requested ldbsearch result of test provision
Comment 19 Matthieu Patou 2011-06-05 14:31:24 UTC
Ok I found a small problem and released a fix to deal with the fact that the SD in to be updated provision might not have owner or group.

Can you revert the different change that I asked and try the two following patches.
Comment 20 Matthieu Patou 2011-06-05 14:33:08 UTC
Created attachment 6522 [details]
First patch, fix the origin of refsddl
Comment 21 Matthieu Patou 2011-06-05 14:33:57 UTC
Created attachment 6523 [details]
Second patch, accept more variety of SD
Comment 22 Dirk Pauli 2011-06-10 21:30:42 UTC
Created attachment 6564 [details]
debug output after applying patches

after reverting the first two changes and applying the 2 new patches, the script is running far longer, but still failing (with a different error). see attached logfile dbg3.txt
Comment 23 Matthieu Patou 2011-06-11 13:00:11 UTC
Created attachment 6566 [details]
patch for last problem
Comment 24 Matthieu Patou 2011-06-11 13:01:07 UTC
Hi Dirk,

Someone introduced a regression lately, the last patch should fix it. Please rebuild your samba before trying upgraderprovison
Comment 25 Dirk Pauli 2011-06-11 20:26:14 UTC
Looks like it is working now!
Thank you - from my perspective, this Bug can be closed after publishing the patches.

Comment 26 Matthias Dieter Wallnöfer 2011-06-20 12:21:06 UTC
According to http://gitweb.samba.org/?p=samba.git;a=commitdiff;h=5db07d2f42e6bbc0023a504f30b9dcc8fd31b230 this should have been fixed.
Comment 27 Matthieu Patou 2011-06-20 13:01:11 UTC

Your issue has forced me to go back to upgradeprovision, I released something like 30 patches for it today, so chances are that your provision is not 100% correct :-).

Especially I found problems with the calculation of the group of mostly each security descriptor.

Would be safe to upgrade to current git and rerun upgradeprovision.