It didn't let me post the log, please see URL.  Also, I accidentally did those 
wbinfo's on the wrong server, here they are for the correct server:

# wbinfo -n "TCS_MAIN_DOM\Domain Admins"
S-1-5-21-2020293289-429224891-1907648334-512 Domain Group (2)

# wbinfo -Y S-1-5-21-2020293289-429224891-1907648334-512
10000

# wbinfo -r "TCS_MAIN_DOM\jarboed"
10001
10002
10003
10004
10005
10006
10007
10008
10009
10010
10011
10012
10013
10014
10000

The perspective from the CUPS side:

Originally submitted "smbprn.00000015 Test Page" is Job # 12.

D [26/Jan/2004:14:57:33 -0500] AcceptClient() 4 from localhost:631.
D [26/Jan/2004:14:57:33 -0500] ReadClient() 4 POST /printers/TC5-EAST-TWO 
HTTP/1.1
D [26/Jan/2004:14:57:33 -0500] print_job: auto-typing file...
D [26/Jan/2004:14:57:33 -0500] print_job: request file type is 
application/vnd.cups-raw.
D [26/Jan/2004:14:57:33 -0500] check_quotas: requesting-user-name = 'jarboed'
D [26/Jan/2004:14:57:33 -0500] print_job: requesting-user-name = 'jarboed'
D [26/Jan/2004:14:57:33 -0500] Adding default job-sheets values "tcs,none"...
I [26/Jan/2004:14:57:33 -0500] Adding start banner page "tcs" to job 12.
I [26/Jan/2004:14:57:33 -0500] Adding end banner page "none" to job 12.
I [26/Jan/2004:14:57:33 -0500] Job 12 queued on 'TC5-EAST-TWO' by 'jarboed'.
D [26/Jan/2004:14:57:33 -0500] Job 12 hold_until = 0
D [26/Jan/2004:14:57:33 -0500] ProcessIPPRequest: 4 status_code=0
D [26/Jan/2004:14:57:33 -0500] CloseClient() 4

Job samba attempted to cancel was #9.  This fails because 9's state is already 
cancelled.  Samba interprets this as Access Denied.

D [26/Jan/2004:15:01:32 -0500] AcceptClient() 4 from localhost:631.
D [26/Jan/2004:15:01:32 -0500] ReadClient() 4 POST /jobs HTTP/1.1
E [26/Jan/2004:15:01:32 -0500] cancel_job: job id 9 is cancelled - can't 
cancel!
D [26/Jan/2004:15:01:32 -0500] Sending error: client-error-not-possible
D [26/Jan/2004:15:01:32 -0500] ProcessIPPRequest: 4 status_code=404
D [26/Jan/2004:15:01:32 -0500] CloseClient() 4

Instead of #12, samba paused Job #9.  At this point "smbprn.00000012 Test 
Page" (#9) appears in the queue.

D [26/Jan/2004:15:04:00 -0500] AcceptClient() 6 from localhost:631.
D [26/Jan/2004:15:04:00 -0500] ReadClient() 6 POST /jobs HTTP/1.1
D [26/Jan/2004:15:04:00 -0500] HoldJob: id = 9
D [26/Jan/2004:15:04:00 -0500] SetJobHoldUntil(9, "indefinite")
D [26/Jan/2004:15:04:00 -0500] SetJobHoldUntil: hold_until = 0
I [26/Jan/2004:15:04:00 -0500] Job 9 was held by 'jarboed'.
D [26/Jan/2004:15:04:00 -0500] ProcessIPPRequest: 6 status_code=0
D [26/Jan/2004:15:04:00 -0500] CloseClient() 6

Here samba cancels #9 instead of #12.

D [26/Jan/2004:15:05:02 -0500] AcceptClient() 6 from localhost:631.
D [26/Jan/2004:15:05:02 -0500] ReadClient() 6 POST /jobs HTTP/1.1
D [26/Jan/2004:15:05:02 -0500] CancelJob: id = 9
I [26/Jan/2004:15:05:02 -0500] Job 9 was cancelled by 'jarboed'.
D [26/Jan/2004:15:05:02 -0500] ProcessIPPRequest: 6 status_code=0
D [26/Jan/2004:15:05:02 -0500] CloseClient() 6

For the last cancel, when samba returns "Error Processing Command", I don't 
see anything in the cups logs, I think because of this error:

[2004/01/26 15:06:03, 5] rpc_parse/parse_prs.c:prs_werror(695)
      0000 status: WERR_INVALID_PRINTER_NAME


In the logs I see little-endian messages from rpc_server/srv_pipe_hnd.c ... 
are these okay for big-endian machine?

The link in the comment above left of the tilde at the end.  Should be <a 
href="http://home.tampabay.rr.com/djarboe/bc006784.log~">http://home.tampabay.r
r.com/djarboe/bc006784.log~</a>

I need level 10 debug logs of the failures.  Thanks.

Too large to attach, it is available at the URL listed in the summary.  
Bugzilla kept mangling the url because it endsd in a tilde, but I can try a 
few more things again:

http://home.tampabay.rr.com/djarboe/bc006784.log%7E

<pre>http://home.tampabay.rr.com/djarboe/bc006784.log~</pre>

14:56:30, XP client selects start/settings/Printers & Faxes
14:57:00, XP client selects properties for printer TC5-EAST-TWO share on linps1
14:57:30, Print test page, "OK"
14:58:00, Close Properties for printer
14:58:30, Open printer window, see "smbprn.00000015 Test Page" in queue.
14:59:00, Refresh, see "smbprn.00000015 Test Page" in queue.
14:59:30, close printer window.
15:00:00, Refresh printers, Documents for [TC5-EAST-TWO] has 
always remained at 0, though it should be 1.  This behavior 
does not change, even when many documents are in the queue.

++ This works ok for me.

15:00:30, Refresh printers, Documents still at 0.
15:01:00, Open TC5-EAST-TWO printer, see "smbprn.00000015 Test Page" in queue.
15:01:30, attempt to cancel "smbprn.00000015 Test Page", get "Access 
Denied" though I should be a printer admin, and though I am the 
owner of the job.

++ Can't reproduce this either.  Jobs are successful deleted.

Still looking at the other issues.  I would suggest removing 
the '@' from the printer admin line since you are not using 
netgroups.

After some more invesigation, I'm pretty sure this is fixed 
in 3.0.2a.  See bug 770 for details and the WHATSNEWS in the 
current release.

Jerry, rebuilt the 3.0.2a SLES8 src rpm from sernet.de, removed all tdb's 
except secrets and started fresh, these problems persist.

> 15:00:00, Refresh printers, Documents for [TC5-EAST-TWO] has 
> always remained at 0, though it should be 1.  This behavior 
> does not change, even when many documents are in the queue.
> 
> ++ This works ok for me.

Documents count is always 0... can you see this in the log?

> 15:01:30, attempt to cancel "smbprn.00000015 Test Page", get "Access 
> Denied" though I should be a printer admin, and though I am the 
> owner of the job.
>
> ++ Can't reproduce this either.  Jobs are successful deleted.

This continues to fail for me... cups still reporting operations being 
performed on the wrong job jumber.

> Still looking at the other issues.  I would suggest removing
> the '@' from the printer admin line since you are not using
> netgroups.

I changed this as you suggested before running these tests again.

There was one more printing bug fixed port 3.0.2a.
Try adding the patch posted in bug 1147 and let me 
know.

Thanks for the suggestion, I rebuilt and retried with that patch, but the 
problem persists.

This job # thing is weird, I printed a test page and cups received it as job 
3741.  Now when samba tries to do anything with it, it's using job 3242.  And 
the number of documents, though 1 right after the job was submitted, goes back 
to 0 again and stays there (this is with queue paused to make testing this 
easier).

Created attachment 440 [details]
fix byte ordering bug when storing updating pid in print_queue_update()

Are there any panics in the log ?  

Is there any way you could test 3.0.2a (rather than 
just patching 3.0.1)?  (note that the patch in comment #11
and the one for bug 1147 are post 3.0.2a patches).

s390 is a big-endian arch correct?  If so then the 
attached patch might help.

> Are there any panics in the log ?

No panics during these tests, not even before the (bug 1147) &cgdata.dptr 
patch.

> Is there any way you could test 3.0.2a (rather than 
> just patching 3.0.1)?  (note that the patch in comment #11
> and the one for bug 1147 are post 3.0.2a patches).

I've been using 3.0.2a recently, and both patches were tried with 3.0.2a, not 
3.0.1.

> s390 is a big-endian arch correct?  If so then the
> attached patch might help.

Yes s390 is big-endian.  The SIVAL patch may have helped something, but things 
still fail.  Documents still shows 0, and samba still performs operations on 
phantom job numbers.

Just now, according to lpq:
1st     jarboed 3743    smbprn.00000001 Untitled - Note 8192 bytes

But the debug log of cups shows Samba tried to cancel a different job #:
D [15/Mar/2004:14:12:46 -0500] CancelJob: id = 3244

Thank you for looking at this, your SIVAL patch was very encouraging.

Would the contents of any of the following keys in that printing tdb be 
helpful?

INFO/linear_queue_array
STATUS/TC5-EAST-TWO
NOTIFY_PID_LIST
INFO/jobs_changed
INFO/version
(a 4 byte key 00 00 00 01)
CACHE/TC5-EAST-TWO
INFO/total_jobs
INFO/nextjob

The printing code is still over my head :(

~ Daniel

Jerry, I looked at the packed structure in the tdb record with key 00 00 00 
01, and all the good stuff looks like its in little endian.

pid: little endian
sysjob: little endian
FD: FFFFFFFF
starttime: little endian
status: 00000000
size: little endian
page count: little endian
spooled: little endian
smbjob: little endian

Should these values be stored in the same endian encoding to match the 
platform, or is it correct as is?

tdb's should be endian safe. 

Created attachment 443 [details]
Here's another patch that fixes some print change notify bugs.

> tdb's should be endian safe. 

You are right of course.  Adding a DEBUG to printing.c:unpack_pjob,
pjob->pid, sysjob, smbjob, etc shows they all end up correct.

> Here's another patch that fixes some print change notify bugs.

Thank you, I applied this one on top of the others, here's what happens:

When I pause the queue, an extra printer icon appears at the bottom with no 
corresponding text though without the little hand underneath (from notify?).  
This also happened before this last patch.  The Status still shows Ready.

Refreshing causes the status to switch to Paused, and the "extra" printer icon 
to disappear.

Sent the printer two jobs, Documents shows 0, even after refreshing.  The 
queue shows the correct information, though, 2 document(s) in queue.  This is 
better than we had in 3.0.0 times.

Deleting "smbprn.00000006 Unititled - Notepad" prompts "Are you sure you want 
to cancel the selected print jobs?"

Yes results in "Error processing commmand." message in the right side of the 
bottom status bar (next to "2 document(s) in queue").

This time, however, from the cups debug2 log it doesn't look like it received 
anything (unlike before this patch, where it would receive a request for a 
cancel operation on the wrong job #).

Is there a list of operations you'd like me to perform at a level 10 log with 
these patches, or do you still have enough to work with?

Thank you,
~ Daniel

Daniel,  I think I'm going to need a level10 debug log from 
you performing the operations you describe.  The cups debug 
log would probably help as well.  Thanks.

Created new level 10 log with Samba 3.0.3pre1 
http://home.tampabay.rr.com/djarboe/3.0.3pre1.log for the following operations:

13:17:00, open "Printers and Faxes" on server (everything shows "Ready").
13:17:30, pause queue TC5-EAST-TWO, status still shows "Ready" but a grayish 
printer icon appears underneath the list of printers, with nothing in the 
Name, Documents, Satus, Comments, Location, or Model columns.  The icon is 
different from the other printer icons, which have the "shared" hands 
underneath.  Cups log shows pause happened at 13:17:31
13:18:00, refresh.  The mysterious icon dissapears and now status does 
show "Paused."
13:18:30ish-13:19:15ish, print two small documents from notepad.  Cups access 
log reports the HTTP POSTs at 13:18:59 and 13:19:13, CUPS jobs 3757 and 3758.  
Also, there is no printer icon that appears in the lower right of the task bar 
telling me about print jobs I have pending.
13:20:00 Refresh, Documents still shows 0 though there are two waiting in the 
queue.
13:20:30 Open printer window, see the two smbprn jobs and "2 document(s) in 
queue"
13:21:00 Attempt to remove a job (second one), get "Error processing 
command."  There is nothing in the cups logs for this operation, though in 
previous versions of Samba on this platform, CUPS would report that samba 
tried to delete job xxxx where xxxx was an older job number.

Anyway, the level 10 log is now available at 
http://home.tampabay.rr.com/djarboe/3.0.3pre1.log.  This time the cups log was 
not particularly helpful, because it appears with the recent patches that 
Samba never tells CUPS anything about cancelling a job, which may be arguably 
better than trying to delete the wrong job.

I'd like to make sure it's clear that the queue stuff has never worked quite 
right in 3.0.x for this platform.  Your patches often change the exhibited 
behavior, though.  The queue stuff seemed to work okay back in 2.2.8a (except 
for some of the notify stuff).  This is a WinXP client, but W2K fails 
similarly.

Daniel, could you get me a failure log against 3.0.3 as well ?  
Including printing the job from a Windows client ?  Thanks.

Created attachment 492 [details]
Log for 3.0.3

Here is the same thing on 3.0.3.  I didn't see a SLES8 3.0.3 srpm on sernet.de
or GD at SuSE, so I dropped the 3.0.3 source into the Volker Lendecke (sernet)
3.0.3rc1 SPEC (Without %{name}-20040421.diff patch and with make_afsstuff 0).
Hope this is ok.

Times should be within a second or two.

I start out with these two windows already open on the client:
The client's Printers and Faxes window (with the installed printer).
Printers and Faxes window on the print server.

Will also attach a PNG of the windows client afterwards, so you can see what I
mean with these "phantom" icons.

8:46:00, pause queue TC5-EAST-ONE, status still shows "Ready" but a greyish
printer icon appears underneath the list of printers, with nothing in the Name,

Documents, Satus, Comments, Location, or Model columns.  The icon is different
from the other printer icons, which have the "shared" hands underneath.

The client's printers and faxes window correctly shows paused

8:46:30, refresh server printers and faxes window.  The "phantom" icon
dissapears and now status correctly shows "Paused."

8:47:00, print two small documents from notepad.  There is no printer icon that

appears in the lower right of the task bar telling me about print jobs I have
pending.
Client printers and faxes window shows: 0 documents
Server printers and faxes windows shows: 0 documents

Print jobs are cups jobid 3846 and 3847 (smbprn.00000007 and smbprn.00000008).

8:47:30 refresh client's printers and faxes window, shows: 0 documents
8:48:00 Refresh server Printers/Faxes, shows: 0 Documents

8:48:30 Open printer queue window (via client's printers and faxes window), see

the two smbprn jobs and "2 document(s) in queue"

8:49:00 refresh client's printers and faxes window, shows: 0 documents
8:49:30 refresh server printer/faxes documents, still shows 0

8:50:00 Attempt to remove a job (second one), get "Error processing command."

8:50:30, select "resume printing" for the samba share from the server printers
and faxes window.  Get that same icon thing, and status remains paused.
The client pritners/faxes window shows ready (0 documents).
Print queue window has a new "phantom" icon, and shows "3 document(s) in
queue".
See samba-3.0.3.png (from WinXP client) long after the jobs had finished.

Created attachment 493 [details]
Screencap of aftermath on WinXP client

This shows the "phantom" icons I was talking about, presumably print notify
problems?  The jobs had completed a few minutes ago, printer TC5-EAST-TWO on
linps2.

Created attachment 496 [details]
smb.conf for 3.0.3, please ignore previous smb.conf in comments above

Since the original report I've changed some settings in smb.conf.  Here is the
smb.conf used for the 3.0.3 log attachment

Daniel, with 1279 fixes I've put in several endian fixes, and I've definitely
fixed my 390 problems, which seem to overlap with many of yours, so can you
please retest now?  the last change I put in was changeset 560.

Tried against 3.0.4 (post changeset 560).  The biggest problems are fixed now 
(thanks!), but some still remain.

For example, When pausing the queue, the "phantom" icon still appears on the 
server Printers and Faxes dialog and the status still shows Ready (though the 
client Printers and Faxes correctly shows paused).  Also, after sending a 
couple of print jobs, documents still shows 0, and there is no pending 
documents for [username] printer icon on the lower right until I manually 
refresh one of the Printers and Faxes windows.  As documents print, the number 
of documents does not change unless the Printers and Faxes window is manually 
refreshed.  So there are still some issues which will affect 
monitoring/managing these printers.

Will work on getting a level 10 debug for 3.0.4 ...

Created attachment 504 [details]
Log for 3.0.4

Samba 3.0.4 level 10 machine log, Windows XP client

Approximate times:

11:00:00 - Pause queue via server's Printers and Faxes window... phantom icon
appears at the bottom of the shared printer list, and status still shows Ready.

However, the status of the client's Printers and Faxes window correctly says
paused.

11:00:30 - Refresh server's Printers and Faxes window, because I will
eventually need to "Resume Printing" the paused queue, but Windows thought it
was still Ready (and only showed option for paused, not resume).  After
refresh, server's Printers and Faxes window correctly shows the printer
Paused, and the phantom icon is gone.

11:01:00 - Send two jobs, both client's and server's Printers and Faxes windows

only show 0 documents.	No "pending documents" icon on lower left of client's
taskbar.

11:04:00 - Refresh server printers and Faxes.  Shows 2 documents now, but
client's still shows 0, and no pending on the client's taskbar yet.

11:04:30 - Refresh client's printers and faxes.  Shows 2 documents now, but
still no pending documents on the client's taskbar (though I've seen it appear
sometimes with a Printers and Faxes window refresh and Samba 3.0.4).  This time

it didn't for some reason.

11:05:00 - Open queue, see my two jobs.  Still no pending documents on client's

taskbar.

11:05:30 - Resume Printing from server's Printers and Faxes windows.  Get a new

phantom icon again at the bottom of the shared printers list, and status still
shows paused.  Client's printers/faxes window shows Ready.  Status of first job

in printer queue window shows printing.

11:06:30ish sometime around this time, both jobs simultaneously disapear from
the client's queue listing

11:09:00 both client's and server's Printers/Faxes window, documents still show

2.  Refresh client's printers/faxes window, now shows 0.  Server's still shows
2, and paused, with phantom icon.

11:09:30 refresh server's printers/faxes window, now shows 0 documents and
Ready and everything is fine again.

Daniel, where do we sit with this in 3.0.6 ?  
Also, if the bug still exists, please update the 
hardware field. 

Having winbindd issues on 3.0.6 right now, but last I tried with 3.0.5rc1 the 
print problems still persisted.  I've heard nothing recently from IBM on 
this.  I will go through a full debug level 10 log with next week's 3.0.7 
release, or sooner if I can convince the 3.0.6 installation to let me log on.  
Our other print server is still running 3.0.2a with printing patches from 
bugzilla 1007 and 1147 and svn revision 560.

Thanks,
~ Daniel

Make sure you get the winbind patch from 
http://samba.org/~jerry/patches/post-3.0.6/
if applicable.  

Thanks, that patch got me working again.  Everything is the same from 3.0.4.  
Do you want me to generate new level 10 logs?

~ Daniel

That would be helpful.  Thanks Daniel.

Created attachment 622 [details]
debug level 10 against 3.0.6

Timestamps for client actions are at the top of the log.
Very similar to 3.0.4 log

I can confirm strange behaviour - the problem is that it's rather stochastic.
Strangely enough, I didn't notice any problems until version 3.0.5 or so, but
what is described here resembles my observations.

It seems that for some reason at some point the cups routines in print_cups.c
cease being called; suddenly there are no more debug printouts in log.smbd.

Running smbcontrol smbd reload-config suffices to restore order.

Continuing investigation...

Oh, and my Samba version is 3.0.7-1 Debian.
The cupsys version is 1.1.20final+rc1-6, but that seems irrelevant. I think it's
pretty clear that the bug isn't in the cups backend.

OK.  Lots of working on printing for 3.0.11 (although the bug 
still might exist on solaris).  Please test 3.0.11 once it 
is released tomorrow.  Reopen if necessary.

sorry for the same, cleaning up the database to prevent unecessary reopens of bugs.