Created attachment 7516 [details] lp_cache_locked_write_times.patch against git-latest Under certain circumstances, Samba, reflecting the behaviour of Windows, caches write times. This patch adds a share-level option to turn that behaviour off. Usage: cache locked write times = yes|no
Created attachment 7517 [details] lp_cache_locked_write_times.patch against Samba 3.6
Why would you want such functionality?
(In reply to comment #2) > Why would you want such functionality? Hi Volker, I've enhanced the media_harmony VFS module to allow clients to control the mtime they see on specific folders in order to deal with shared-folder problems with the Avid media-editing application. The caching of write times defeats that behaviour, and my enhancements to media_harmony cannot work with it turned on. The patch defaults to Samba's regular write caching behaviour, so normal users wouldn't see any change. Links for more information on the media_harmony VFS module and Avid: http://ingex.sourceforge.net/MediaHarmony/ https://lists.samba.org/archive/samba-technical/2012-March/082332.html https://bugzilla.samba.org/show_bug.cgi?id=8841 http://www.avid.com/ Thanks. Andrew
(In reply to comment #3) > I've enhanced the media_harmony VFS module to allow clients to control the > mtime they see on specific folders in order to deal with shared-folder problems > with the Avid media-editing application. The caching of write times defeats > that behaviour, and my enhancements to media_harmony cannot work with it turned > on. Hmm. I wonder if we can't do that cleanly behind the VFS so that the module can determine that behaviour. Can't you cheat and write your version of mtime into locking.tdb from within your module? It might be too slow, but depending on your exact requirements it might not be necessary to do that on every write. Volker
(In reply to comment #4) > (In reply to comment #3) > > I've enhanced the media_harmony VFS module to allow clients to control the > > mtime they see on specific folders in order to deal with shared-folder problems > > with the Avid media-editing application. The caching of write times defeats > > that behaviour, and my enhancements to media_harmony cannot work with it turned > > on. > > Hmm. I wonder if we can't do that cleanly behind the VFS so that the module can > determine that behaviour. Can't you cheat and write your version of mtime into > locking.tdb from within your module? It might be too slow, but depending on > your exact requirements it might not be necessary to do that on every write. Interesting idea. One concern: Does each client get their own version of the locking.tdb file, or their own mtimes within it that other clients don't get? In my case, the essential part of having the clients control their own mtime is that each client needs to get a different, client-specific mtime for the same directory. If they don't get that, the applications constantly interrupt each other and they get little work done. Andrew
(In reply to comment #5) > Interesting idea. One concern: Does each client get their own version of the > locking.tdb file, or their own mtimes within it that other clients don't get? > In my case, the essential part of having the clients control their own mtime is > that each client needs to get a different, client-specific mtime for the same > directory. If they don't get that, the applications constantly interrupt each > other and they get little work done. Wait a second -- you are presenting different mtimes to different clients? Wow. I haven't read your module comments in detail: Why do you have to do that?? Volker
(In reply to comment #6) > Wait a second -- you are presenting different mtimes to different clients? Wow. > I haven't read your module comments in detail: Why do you have to do that?? Whenever Avid sees an mtime update on a media directory, it automatically launches a database refresh than interrupts the user for anywhere from a few seconds to a few minutes. So if one editor is bringing in new material - and thus constantly updating the directory's mtime - another editor who's trying to do a playback session will be constantly interrupted. Giving each editor their own mtime allows them to decide exactly when they want to be interrupted for a database refresh, independent of what other editors are doing. Andrew
Created attachment 7522 [details] lp_cache_locked_write_times.patch against git-latest Add "cache locked write times" parameter. Supersedes previous patch. Compared to previous patch: - Adds manpage entry. - Brings up-to-date with git-latest. Andrew Klaassen
looks like the media_harmony module is the only valid use case for the "cache locked write times". So, are the objections to adding this parameter?
metze's idea was to introduce something like "bool write_time_cacheable" in the vfs files_struct, which defaults to yes, media_harmony could set this to no accordingly.
(In reply to comment #10) > metze's idea was to introduce something like "bool write_time_cacheable" in the > vfs files_struct, which defaults to yes, media_harmony could set this to no > accordingly. Hi all, if I get you right, with the proposed solution one would be forced to not have samba caching the write time if using the media harmony vfs. So it would not be configurable at all. There might also be use cases where you want to use a media harmony share with cachable write times, e.g. if you have a share that is used for writing by just a single avid client and other avid clients are just reading from that share (except their individual msFMID.pmr and msmMMOB.mdb files). In this case you could keep avids default behaviour of refreshing the database with the benefit of not having to manually force it each time you write. I like the idea of making it easy to use and maintain. But even more I would like being configurable, because the above example is one way I'm planning to use it. ;-) Regards Michael
(In reply to comment #11) > (In reply to comment #10) > > metze's idea was to introduce something like "bool write_time_cacheable" in the > > vfs files_struct, which defaults to yes, media_harmony could set this to no > > accordingly. > > Hi all, > > if I get you right, with the proposed solution one would be forced to not have > samba caching the write time if using the media harmony vfs. > So it would not be configurable at all. > > There might also be use cases where you want to use a media harmony share with > cachable write times, e.g. if you have a share that is used for writing by just > a single avid client and other avid clients are just reading from that share > (except their individual msFMID.pmr and msmMMOB.mdb files). In this case you > could keep avids default behaviour of refreshing the database with the benefit > of not having to manually force it each time you write. > > I like the idea of making it easy to use and maintain. But even more I would > like being configurable, because the above example is one way I'm planning to > use it. ;-) I'm afraid the behaviour is somewhat more complicated than that, in a way that makes it sound like you don't need this patch at all. :-) - In the default Samba configuration, the mtime that's returned for a client depends on things like who else is browsing the directory. This is to match Windows behaviour. - To get media_harmony to match that behaviour - i.e. act like it previously did, which should more-or-less match the behaviour you're describing - you don't have to do anything special; just enable media_harmony and use it. This patch isn't needed for that. - To get my new media_harmony mtime behaviour - which we've found critical for shared workflow with multiple people doing ingest and editing - you need both this patch and manual (or scripted - see the example script I've included) creation of "fake mtime" files for media_harmony to get its per-client, per-directory mtimes from. If Samba does go the alternate route (i.e. modify the vfs_files_struct), I should be able to provide a media_harmony-specific option that allows the behaviour to be turned on or off. Andrew
(In reply to comment #12) > - To get media_harmony to match that behaviour - i.e. act like it previously > did, which should more-or-less match the behaviour you're describing - you > don't have to do anything special; just enable media_harmony and use it. This > patch isn't needed for that. > > ... > > - To get my new media_harmony mtime behaviour - which we've found critical for > shared workflow with multiple people doing ingest and editing - you need both > this patch and manual (or scripted - see the example script I've included) > creation of "fake mtime" files for media_harmony to get its per-client, > per-directory mtimes from. Hi Andrew, my post was directed towards the proposal in comment 10. Not towards your patch at all, which I find to be a great solution. I'm aware of the difference between both scenarios and willing to use both. That's why I wanted to show that it's good to have this configurable per share (as it already is with your patch). Michael
(In reply to comment #12) > If Samba does go the alternate route (i.e. modify the vfs_files_struct), I > should be able to provide a media_harmony-specific option that allows the > behaviour to be turned on or off. Exactly, this should be an option of the media_harmony module not a generic samba option.
Andrew, would you like to come up with a patch for this? 4.0 rc1 is coming soon and then it might be difficult to get that change in before 4.0.
(In reply to comment #15) > Andrew, would you like to come up with a patch for this? 4.0 rc1 is coming soon > and then it might be difficult to get that change in before 4.0. Hi Björn, My comfort level at this point unfortunately ends with the per-share patch I originally submitted. I don't trust my ability to not blow everything up when messing with the internals of the VFS code. :-)
Created attachment 8971 [details] alternative approach to handle media files/databases Sorry, if this is a wrong place for my request. The approach of renaming .pmr and .mdb database files works well. Otherwise - Avid Unity and third party software developers are using another method: client based masking and redirecting of whole media directories. This is more comfortable, as the each client has his own active media directory to write, and can also read direcotories created by another clients. No extra refreshing actions are needed. By Avid and others it is implemented with client-side applications/virtual file systems - but we have samba vfs. In advance, my programming skills are very poor, somehow i was able to change the code of mediaharmony to partially fullfill desired beahaviour (as decribed below) and compile it (probably, my code is for courageous natures only and needs rework). It works so far, except instead of using clients NETBIOS names they are replaced with IP adresses. Here my understanding how Avid media access works: Avid working media directory is Avid MediaFiles/MXF/1/ (on top of share/disc). This means that all created media files (imported,captured, rendered) will be written in this directory. Application is constantly watching this working directory for changes. On each change ( added new files, deleted files), new media files scan is starting and databases will be refreshed. (Avid is using media databases (msmFMID.pmr, msmMMOB.mdb), one for each media directory.) Avid is also looking (only on start or manual refresh?) in other numbered subdirectories (e.g. Avid MediaFiles/MXF/2/ or Avid MediaFiles/MXF/whatever.77/ ) for databases. They will be opened in read-only mode an containing files can be used for editing. In this way can the application read files created by other users/workstations. Howewer, for write-access is Avid using only “Avid MediaFiles/MXF/1/”, and only databases and files in this folder will be scanned and actualised. What`s the job for vfs: Any call from WorkstationNameX to “Avid MediaFiles/MXF/1/” and underlaying folder/files should be redirected to “Avid MediaFiles/MXF/WorkstationNameX.01/...” The same should be done also for directory “OMFI MediaFiles/01/” (without “MXF” subfolder) - also any call from WorkstationNameX to “OMFI MediaFiles/01/” with related underlaying folder and files should be redirected to “OMFI MediaFiles/WorkstationNameX.01/...” If user manually renames “Avid MediaFiles/MXF/WorkstationNameX.01/” to “Avid MediaFiles/MXF/WorkstationNameX.02/”, renamed directory should be handled as any other numbered media directory with read only acess. Structure expected sides Avid client application: Avid MediaFiles/MXF/01 msmFMID.pmr msmMMOB.mdb abc.mxf <dir>Creating Avid MediaFiles/MXF/02(read only) msmFMID.pmr msmMMOB.mdb xyz.mxf Creating tempfiles OMFI MediaFiles/01 msmFMID.pmr msmMMOB.mdb abcdef.mxf <dir>Creating OMFI MediaFiles/02(read only) msmFMID.pmr msmMMOB.mdb xyzuw.mxf <dir>Creating Desired Vfs functionality: Avid MediaFiles/MXF/(01 (seen by client)->WorkstationNameA.01(server side)) msmFMID.pmr msmMMOB.mdb abc.mxf <dir>Creating Avid MediaFiles/MXF/(02(seen by client)->WorkstationNameB.01(server side)) msmFMID.pmr msmMMOB.mdb xyz.mxf <dir>Creating OMFI MediaFiles/01->WorkstationNameA.01 msmFMID.pmr msmMMOB.mdb abcde.mxf <dir>Creating OMFI MediaFiles/02->WorkstationNameB.01 msmFMID.pmr msmMMOB.mdb xyzuw.mxf <dir>Creating Desired resulting directory structure on share: Avid MediaFiles/MXF/WorkstationNameA.01 msmFMID.pmr msmMMOB.mdb abc.mxf <dir>Creating Avid MediaFiles/MXF/WorkstationNameB.01 msmFMID.pmr msmMMOB.mdb xyz.mxf <dir>Creating OMFI MediaFiles/WorkstationNameA.01 msmFMID.pmr msmMMOB.mdb abcde.mxf <dir>Creating OMFI MediaFiles/WorkstationNameB.01 msmFMID.pmr msmMMOB.mdb xyzuw.mxf <dir>Creating Desired directory structure seen by client: Avid MediaFiles/MXF/01 (own files, writable) Avid MediaFiles/MXF/WorkstationNameB.01 (readonly) OMFI MediaFiles/01 (own files, writable) OMFI MediaFiles/WorkstationNameB.01 (read only) What do you think about it? Many thanks, Milos
Comment on attachment 8971 [details] alternative approach to handle media files/databases Would it be possible to add this as a unified diff? "diff -u" is more readable than the current patch. Thanks.
--- On Sun, 6/16/13, samba-bugs@samba.org <samba-bugs@samba.org> wrote: > Here my understanding how Avid media access works: > > Avid working media directory is Avid MediaFiles/MXF/1/ > (on top of share/disc). > This means that all created media files (imported,captured, > rendered) will be > written in this directory. > Application is constantly watching this working > directory for changes. On each > change ( added new files, deleted files), new media files > scan is starting and > databases will be refreshed. (Avid is using media databases > (msmFMID.pmr, > msmMMOB.mdb), one for each media directory.) > Avid is also looking (only on start or manual > refresh?) in other numbered > subdirectories (e.g. Avid MediaFiles/MXF/2/ or Avid > MediaFiles/MXF/whatever.77/ > ) for databases. They will be opened in read-only mode an > containing files can > be used for editing. In this way can the application read > files created by > other users/workstations. > > Howewer, for write-access is Avid using only “Avid > MediaFiles/MXF/1/”, and > only databases and files in this folder will be scanned and > actualised. I'm no Avid expert, but my experience is different. Avid only writes the first 5000 or so files to /1/; then it creates /2/ and starts writing there, and after another 5000 or so files creates /3/, etc. It scans and writes indexes to all media subdirectories that it sees. (Or does making some subdirectories read-only via Samba change this behaviour?) > What`s the job for vfs: > Any call from WorkstationNameX to “Avid > MediaFiles/MXF/1/” and underlaying > folder/files should be redirected to “Avid > MediaFiles/MXF/WorkstationNameX.01/...” > > The same should be done also for directory “OMFI > MediaFiles/01/” (without “MXF” > subfolder) - > also any call from WorkstationNameX to “OMFI > MediaFiles/01/” with related > underlaying folder and files should be redirected to > “OMFI > MediaFiles/WorkstationNameX.01/...” > > If user manually renames “Avid > MediaFiles/MXF/WorkstationNameX.01/” to “Avid > MediaFiles/MXF/WorkstationNameX.02/”, renamed directory > should be handled as > any other numbered media directory with read only acess. > > > Structure expected sides Avid client application: > > Avid MediaFiles/MXF/01 > msmFMID.pmr > msmMMOB.mdb > abc.mxf > <dir>Creating > > Avid MediaFiles/MXF/02(read only) > > > msmFMID.pmr > msmMMOB.mdb > xyz.mxf > Creating > tempfiles > > OMFI MediaFiles/01 > msmFMID.pmr > msmMMOB.mdb > abcdef.mxf > <dir>Creating > > OMFI MediaFiles/02(read only) > msmFMID.pmr > msmMMOB.mdb > xyzuw.mxf > <dir>Creating > > Desired Vfs functionality: > > Avid MediaFiles/MXF/(01 (seen by > client)->WorkstationNameA.01(server side)) > msmFMID.pmr > msmMMOB.mdb > abc.mxf > <dir>Creating > > Avid MediaFiles/MXF/(02(seen by > client)->WorkstationNameB.01(server side)) > msmFMID.pmr > msmMMOB.mdb > xyz.mxf > <dir>Creating > > OMFI MediaFiles/01->WorkstationNameA.01 > msmFMID.pmr > msmMMOB.mdb > abcde.mxf > <dir>Creating > > OMFI MediaFiles/02->WorkstationNameB.01 > msmFMID.pmr > msmMMOB.mdb > xyzuw.mxf > <dir>Creating > > Desired resulting directory structure on share: > > Avid MediaFiles/MXF/WorkstationNameA.01 > msmFMID.pmr > msmMMOB.mdb > abc.mxf > <dir>Creating > > Avid MediaFiles/MXF/WorkstationNameB.01 > msmFMID.pmr > msmMMOB.mdb > xyz.mxf > <dir>Creating > > OMFI MediaFiles/WorkstationNameA.01 > msmFMID.pmr > msmMMOB.mdb > abcde.mxf > <dir>Creating > > OMFI MediaFiles/WorkstationNameB.01 > msmFMID.pmr > msmMMOB.mdb > xyzuw.mxf > <dir>Creating > > Desired directory structure seen by client: > Avid MediaFiles/MXF/01 (own files, writable) > Avid MediaFiles/MXF/WorkstationNameB.01 (readonly) > OMFI MediaFiles/01 (own files, writable) > OMFI MediaFiles/WorkstationNameB.01 (read only) > > What do you think about it? I don't know enough about Avid's behaviour to know how well it'll work, but it sounds like it's more in line with the emerging standard for sharing Avid media. On the other hand, it's a pretty significant change from the existing working code. You might be interested in discussing this with Steven Umbehocker at OS Nexus (steven.umbehocker at osnexus.com), too, since they've expressed the most interest and done the most work with Mediaharmony since it was released as part of Samba. They're working on integrating it into their appliance, and might be interested in a more standard approach. Here's a bit of what they've done with it so far: http://wiki.osnexus.com/mediawiki/index.php/Avid_Media_Composer%E2%84%A2_%2B_MediaHarmony_Configuration_Guide Andrew
(In reply to comment #19) > I'm no Avid expert, but my experience is different. Avid only writes the first > 5000 or so files to /1/; then it creates /2/ and starts writing there, and > after another 5000 or so files creates /3/, etc. It scans and writes indexes > to all media subdirectories that it sees. (Or does making some subdirectories > read-only via Samba change this behaviour?) Maybe I can clarify this a bit. For the first part it's exactly as Andrew said: 5000 files before Avid begins using the next numeric subdir. But for the second part it's different: Avid only writes indexes (msmMMOB.mdb & msmFMID.pmr) to "<numeric>" subdirs and to sudirs named "<OWNHOSTNAME>.<numeric>". The <numeric> part of the subdir can have at least 8 digits, maybe more. I haven't tested more than 8 digits. MXF-Files from all other subdirs can be read/used in Media Composer if "msmMMOB.mdb" and "msmFMID.pmr" are present in these subdirs. So e.g. assuming the hostname of the scanning Media Composer is "MC1". "MC1" will write msmMMOB.mdb & msmFMID.pmr in subdirs "1", "70", "12345678", "MC1.1", "MC1.20130616", etc. But "MC1" will NOT do so for subdirs "MC2.1", "MC3.20130615" or "Project1". These are treated read-only by "MC1" and can only be used if "msmMMOB.mdb" and "msmFMID.pmr" are already present in these subdirs. (written either by MC1 before renaming the subdir or written by another Media Composer). Michael
Created attachment 8985 [details] unified patch for media harmony (samba 4.0.5) Here are my tests with media_harmony (modified through attached patch) . Server: Fedora18, 64bit, Samba version 4.0.5 Client "192.168.99.201" : Win7 64bit, Media Composer 6.5 Client "192.168.99.204" : Win7 64bit, Media Composer 5.5.3 New share on server created:K-TestShare Server: K-TestShare Client "192.168.99.201": Z:. Client "192.168.99.204": Y:. After importing of first file on client "192.168.99.201": Server K-TestShare `-- Avid\ MediaFiles `-- MXF `-- 1_192.168.98.201.01 |-- Creating |-- msmFMID.pmr |-- msmMMOB.mdb `-- test_file0000.tga1351C16F99.mxf Client "192.168.99.201" Z:. +---Avid MediaFiles +---MXF +---1 │ msmMMOB.mdb │ msmFMID.pmr │ test_file0000.tga1351C16F99.mxf │ └───Creating Client "192.168.99.204" Y:. └───Avid MediaFiles └───MXF └───1_192.168.98.201.01 │ msmMMOB.mdb │ msmFMID.pmr │ test_file0000.tga1351C16F99.mxf │ └───Creating After importing of first file on client "192.168.99.204": Server: K-TestShare `-- Avid\ MediaFiles `-- MXF |-- 1_192.168.98.201.01 | |-- Creating | |-- msmFMID.pmr | |-- msmMMOB.mdb | `-- test_file0000.tga1351C16F99.mxf `-- 1_192.168.98.204.01 |-- Creating |-- msmFMID.pmr |-- msmMMOB.mdb `-- test_file0000.tga1351C176A9.mxf Client "192.168.99.201" Z:. └───Avid MediaFiles └───MXF ├───1_192.168.98.204.01 │ │ msmMMOB.mdb │ │ msmFMID.pmr │ │ test_file0000.tga1351C176A9.mxf │ │ │ └───Creating └───1 │ msmMMOB.mdb │ msmFMID.pmr │ test_file0000.tga1351C16F99.mxf │ └───Creating Client "192.168.99.204" Y:. └───Avid MediaFiles └───MXF ├───1 │ │ msmMMOB.mdb │ │ msmFMID.pmr │ │ test_file0000.tga1351C176A9.mxf │ │ │ └───Creating └───1_192.168.98.201.01 │ msmMMOB.mdb │ msmFMID.pmr │ test_file0000.tga1351C16F99.mxf │ └───Creating Both media files can be accessed on both clients. No issues. We import more files now: about 1.000 still image files, imported on both clients simultaneously. Import works without rescanning messages, without influence between clients. After import are all files accessible on both clients. Mhm, there is an issue: client "201" begins to scan its own media directory. I can not reproduce this behaviour later. We delete now database files (msmMMOB.mdb msmFMID.pmr) in /MXF/1_192.168.98.204.01 (owned by 192.168.98.204) As expected, client "192.168.99.201" does nothing. That's mean, directory /MXF/1_192.168.98.204.01 can be refreshed only from owner - "192.168.99.204". As desired. We take a look at "192.168.99.204". As expected, after deleting of its database files, it begins with refresh and write new database files. As expected. As a hint, refreshes on databases are starting always after switching Media Composer to background - with task manager or whatever. If you are in MC, nothing happens. Now, We import more than 5.000 files on "204". All files added to import are through, we have now 5271 files in "/MXF/1_192.168.98.204.01" (and in one bin). That's OK. But ... 200 remaining files are in automaticaly created "/MXF/2". That's not OK. Server tree now: K-TestShare `-- Avid\ MediaFiles `-- MXF |-- 1_192.168.98.201.01 | `-- Creating |-- 1_192.168.98.204.01 | `-- Creating `-- 2 `-- Creating New created "/MXF/2" is now for both clients "active", and refreshes are performed from both clients. (Media)Harmony is gone. From now, "201" continues writing files in its "/MXF/1", but "204" in "/MXF/2". Each change in "/MXF/2" through "204" calls refreshing on "201" and vice versa. For information, after deleting some files in "204"'s "/MXF/1/", Media Composer persist writing to "/2/". After manualy deleting of "/1/", it (the "/1/") will be from MC automaticaly recreated and flagged as "active" for writing. Possible solution: What we need, is to handle not only pattern "/1/" in middle of the path (code line 117), but also 2,3 and so on, and replace them with client suffixed path. OK, we create (unfortunately) manualy "/MXF/2_192.168.99.204.2" and copy here all files from "/MXF/2/" (including database files). We delete all files in /MXF/2/" to see, if any client can see and relink the files in "/MXF/2_192.168.99.204.2". Inmediately happens nothing - files are offline, but after restart of Media Composer: files are there! On both clients as "read only" directories. Not "active" - ther is no "owner client" for this directory at the moment. This behaviour is desired, and the solution above should work. As said, my programming skills (and my english) are poor and i'm sure, that very long list of "if /1/ then ..., "if /2/ then..." is not the elegant way to solve this. Can anyone help? Thanks, Milos
--- On Wed, 6/19/13, samba-bugs@samba.org <samba-bugs@samba.org> wrote: > Import works without rescanning messages, without influence between > clients. After import are all files accessible on both clients. > Mhm, there is an issue: client "201" begins to scan its own media > directory. I can not reproduce this behaviour later. I found that a lot of subtle scanning weirdness was produced by the fact that Samba tries to emulate Windows server behaviour with caching write times. Have you applied the lp_cache_locked_write_times patch and set: cache locked write times = no If you don't, you won't get consistent scanning behaviour from Avid. Sometimes it'll seem like it works, sometimes it won't. Andrew
(In reply to comment #17) > alternative approach to handle media files/databases > I kind of like your approach, but I think it shouldn't replace media_harmony as it is, as it has a different functionality. Your approach doesn't take in account what media harmony was made for: handling of shared avid folders which are created and filled by a third party application that is not able to create Avid db-files by itself. e.g. ingex (what media harmony vfs was developed for in the first place). In your approach this 'third party subdir' would be ignored by Avid Machines as no avid client would write/update the Avid db-File (mdb+pmr) in it. IMHO your approach should be realised as a separate vfs module. > What`s the job for vfs: > Any call from WorkstationNameX to “Avid MediaFiles/MXF/1/” and underlaying > folder/files should be redirected to “Avid > MediaFiles/MXF/WorkstationNameX.01/...” > > ... > > If user manually renames “Avid MediaFiles/MXF/WorkstationNameX.01/” to “Avid > MediaFiles/MXF/WorkstationNameX.02/”, renamed directory should be handled as > any other numbered media directory with read only acess. > How you implemented it at the moment is kind of a simple hack. IMHO it should work like this: If a client creates any numeric subdir to 'Avid MediaFiles/MXF/' on the share the hostname of this client should be prefixed to the subdirs name on the server. When reading from 'Avid MediaFiles/MXF/' the client should not see it's own hostname prefixes. e.g.: (subdirs of Avid MediaFiles/MXF/) on server seen by client 'HOST1' seen by client 'HOST2' ========= ====================== ==================== 'HOST1.1' '1' 'HOST1.1' 'HOST1.2' '2' 'HOST1.2' 'HOST1.20130618' '20130618' 'HOST1.20130618' 'HOST2.1' 'HOST2.1' '1' 'HOST2.2' 'HOST2.2' '2' 'HOST2.4711' 'HOST2.4711' '4711' 'HOST4.1' 'HOST4.1' 'HOST4.1' 'HOST4.2' 'HOST4.2' 'HOST4.2' 'HOST4.4711' 'HOST4.555' 'HOST4.555' > > What do you think about it? My 2 cents: Like I said: I kinda like your new approach, but I don't think it should replace media_harmony vfs. You should make it's own new vfs module out of it, as it handles things completely different. --- --- P.S.: I like & use media harmony as it is by now and I am happy that it made it's way to the samba release. I'm still subscribed to this bug, as I hope someone will incorporate the 'lp_cache_locked_write_times.patch' in some way/someday. 'til then I will be patching. :-)
(In reply to comment #22) > --- On Wed, 6/19/13, samba-bugs@samba.org <samba-bugs@samba.org> wrote: > > > Import works without rescanning messages, without influence between > > clients. After import are all files accessible on both clients. > > Mhm, there is an issue: client "201" begins to scan its own media > > directory. I can not reproduce this behaviour later. > > I found that a lot of subtle scanning weirdness was produced by the fact that > Samba tries to emulate Windows server behaviour with caching write times. > > Have you applied the lp_cache_locked_write_times patch and set: > > cache locked write times = no > > If you don't, you won't get consistent scanning behaviour from Avid. Sometimes > it'll seem like it works, sometimes it won't. > > Andrew @Andrew, at first many thanks for your work on media_harmony. And for reading. Spontaneous scan happens only once time (shortly after creating new media directories.) - and not in "foreign" directory. This scan took max. 1 second. Modified version works in a different way as your/ingest original in relation to scans/refreshes. Media Composer makes difference between variously numbered directories. As Michael wrote: >So e.g. assuming the hostname of the scanning Media Composer is "MC1". >"MC1" will write msmMMOB.mdb & msmFMID.pmr in subdirs "1", "70", "12345678", >"MC1.1", "MC1.20130616", etc. >But "MC1" will NOT do so for subdirs "MC2.1", "MC3.20130615" or "Project1". >These are treated read-only by "MC1" and can only be used if "msmMMOB.mdb" and >"msmFMID.pmr" are already present in these subdirs. (written either by MC1 >before renaming the subdir or written by another Media Composer). @Michael >How you implemented it at the moment is kind of a simple hack. It's a first C++ code i every seen. And simplest solution is usually the best one. >IMHO it should work like this: >If a client creates any numeric subdir to 'Avid MediaFiles/MXF/' on the share >the hostname of this client should be prefixed to the subdirs name on the >server. >When reading from 'Avid MediaFiles/MXF/' the client should not see it's own >hostname prefixes. >e.g.: (subdirs of Avid MediaFiles/MXF/) >on server seen by client 'HOST1' seen by client 'HOST2' >========= ====================== ==================== >'HOST1.1' '1' 'HOST1.1' >'HOST1.2' '2' 'HOST1.2' >'HOST1.20130618' '20130618' 'HOST1.20130618' >'HOST2.1' 'HOST2.1' '1' >'HOST2.2' 'HOST2.2' '2' >'HOST2.4711' 'HOST2.4711' '4711' >'HOST4.1' 'HOST4.1' 'HOST4.1' >'HOST4.2' 'HOST4.2' 'HOST4.2' >'HOST4.4711' 'HOST4.555' 'HOST4.555' Thanks, it's in principle a confirmation for my tests from today. Modified version does already what you mean. At the moment only for directory "1", and uses IP adresses instead hostnames. This means, you can handle over 5.000 files pro share and workstation. This is more than enough for our site. But sure, it's would be nice to have no limits. We are using modified version since approx. 3 months in our company (5 edit suites, 60 TB Samba Storage) without any crashes or troubles. Combined with ddp from Ardis Technology for project sharing (and also for storage sharing) it's a good and reliable environment for our work. "My" approach are using e.g. Avid Unity, EditShare, ddp, Tiger Technology... I worked , respectively i'm working with this products since more than 12 years. All of them are more or less good and more then less expensive. >Your approach doesn't take in account what media harmony was made for: handling >of shared avid folders which are created and filled by a third party >application that is not able to create Avid db-files by itself. >e.g. ingex (what media harmony vfs was developed for in the first place). >In your approach this 'third party subdir' would be ignored by Avid Machines as >no avid client would write/update the Avid db-File (mdb+pmr) in it. Mhm, i don't understand. This approach does at the end the same as the original. Avoiding conflicts on network shared directories. >IMHO your approach should be realised as a separate vfs module. It would be nice. Many thanks, Milos
--- On Wed, 6/19/13, samba-bugs@samba.org <samba-bugs@samba.org> wrote: > > Your approach doesn't take in account what media harmony > > was made for: handling of shared avid folders which are created > > and filled by a third party application that is not able to create > > Avid db-files by itself. e.g. ingex (what media harmony vfs was > > developed for in the first place). > > In your approach this 'third party subdir' would be ignored by Avid > > Machines as no avid client would write/update the Avid db-File > > (mdb+pmr) in it. > > Mhm, i don't understand. This approach does at the end the same as the > original. Avoiding conflicts on network shared directories. He's referring to this: http://ingex.sourceforge.net/ ...which is where Mediaharmony started and is still used, long before I started working on it. Andrew
(In reply to comment #24) > @Michael > >How you implemented it at the moment is kind of a simple hack. > It's a first C++ code i every seen. And simplest solution is usually the best > one. I didn't mean to your coding style in any way. I'm not a c coder and can not judge this. I did mean this naming scheme of your 1st patch: '01' to 'xxx.02'. And the staticness of just working with the '1' subdirectory, which still remains. No offence, I know it's work in progress. :-) > >Your approach doesn't take in account what media harmony was made for: handling > >of shared avid folders which are created and filled by a third party > >application that is not able to create Avid db-files by itself. > >e.g. ingex (what media harmony vfs was developed for in the first place). > >In your approach this 'third party subdir' would be ignored by Avid Machines as > >no avid client would write/update the Avid db-File (mdb+pmr) in it. > > Mhm, i don't understand. This approach does at the end the same as the > original. Avoiding conflicts on network shared directories. Yes, right, but not only this. The original text says: 'media_harmony - per-client .pmr and .mdb database files so there are no conflicts between Avid editors' Ingex-studio records avid compatible OP-Atom .mxf-files. These files are then transferred to a shared storage. Ingex doesn't create mdb+pmr-files. So in order to use these files an Avid has to scan/index them. This is where media harmony comes in, allowing multiple avids scan/index these subdirs. And this case can't be solved by the new (your) approach. Because e.g. if the host ingex01 creates a subdir '20130624', it would be seen as 'ingex01.20130624' to avid clients and it would not be indexed by any of them. So now you should know what I meant. But I don't think the new (your) approach has to address this problem! I even think you should safely ignore this problem. That's what media_harmony (as it is now) already exists is for. Make your Approach a new seperate vfs module and let the two coexist. And give it a name of it's own, like e.g. 'vfs_avid_share'. So we don't have to talk about your/new/old approach/media_harmony anymore. ;-) Cheers Michael What do you think about moving to a public github repo to allow forking and maybe some collaboration?
> > Mhm, i don't understand. This approach does at the end the same as the > > original. Avoiding conflicts on network shared directories. > > Yes, right, but not only this. > The original text says: 'media_harmony - per-client .pmr and .mdb database > files so there are no conflicts between Avid editors' > Ingex-studio records avid compatible OP-Atom .mxf-files. These files are then > transferred to a shared storage. Ingex doesn't create mdb+pmr-files. So in > order to use these files an Avid has to scan/index them. This is where media > harmony comes in, allowing multiple avids scan/index these subdirs. > And this case can't be solved by the new (your) approach. Because e.g. if the > host ingex01 creates a subdir '20130624', it would be seen as > 'ingex01.20130624' to avid clients and it would not be indexed by any of them. > So now you should know what I meant. Michael, many thanks for your clarifications. The structure of original media_harmony makes for me more sense now. Ok, we can move files created by ingex to media directory of the first Avid client and after indexing are files available for all clients. But for purposes as described (using with external "no Avid" ingest device) is the original approach pretty good designed. > Make your Approach a new seperate vfs module and let the two coexist. > And give it a name of it's own, like e.g. 'vfs_avid_share'. > So we don't have to talk about your/new/old approach/media_harmony anymore. ;-) Conflicts with registered brands and with network shared files should be avoided. ;-) 'vfs_mc_share'? 'vfs_SharedHarmony'? > What do you think about moving to a public github repo to allow forking and > maybe some collaboration? The problem is that samba vfs interface is permanently changing. Code working in this version can be for next samba version unusable (see media_harmony history at ingex source forge - thanks Andrew is it in samba git master now, and will be actualised for each new version). It make sense only if the code is in samba-master (or if anyone is ready and accomplished to adapt it periodically; furthemore enduser must be here able to compile the module for his own system). It is much simplier, if module is provided directly with samba distribution. I can try to take the same way as Andrew, but i'm not sure, if this helps anyone but me :-) "To do" list is in my view not so long: 1, Changing fixed replacing of "/1/" => "/1_<ip.address>.1/" in path to "/<number>/" => "/<hostname>.<number>/" 2, ? Milos
Created attachment 9037 [details] Unityed Media - VFS module for shared avid media access Client "editor01": Server: Avid MediaFiles/MXF/1 ---> Avid MediaFiles/MXF/1_editor01.01 Avid MediaFiles/MXF/7771 ---> Avid MediaFiles/MXF/7771_editor01.7771 Avid MediaFiles/MXF/1/15 ---> Avid MediaFiles/MXF/1_editor01.01/15 Avid MediaFiles/MXF/7MyDir12 ---> Avid MediaFiles/MXF/7MyDir12 Client "editor02": Server: Avid MediaFiles/MXF/1 ---> Avid MediaFiles/MXF/1_editor02.01 Avid MediaFiles/MXF/45 ---> Avid MediaFiles/MXF/45_editor02.45 ... Tested on Windows7 64 bit and Media Composer 6.5 (clients) Samba 4.05, Fedora 18 64 bit (server) Many thanks to Michael and Andrew. Milos
Unityed Media module is now hosted at http://code.google.com/p/vfs-unityed-media/
While I haven't been able to follow exactly all of the bug precisely, I see that the last comment contains the patch for the unityed media module. This went into Samba upstream in 2014 with https://gitlab.com/samba-team/samba/-/commit/15511f7c51686. Closing this bug report. If there are more problems in this area, feel free to re-open.