This will be the second important feature implementation for 1.2.6. SMBTA shall have integrated version control for all components. Some facts about this: -> smbtad can log the VFS modules sub-version number in the database -> smbtad can log it's own version information in the database -> smbtad can log the fact that it is ignoring new common data blocks in the database -> smbtatools can locally store their version information and react on update -> smbtatools can lookup what smbtad is storing -> smbtatools can proactively run the database conversion process if that is required -> all that can be controlled from webSMBTA or smbtaquery I have some thoughts for this. Benjamin, Michael, let us discuss this in a meeting to get the overview, and create a mockup of the different version options. Robert, if you are also interested in this, please say so, we'll find ways, IRC or voIP. I am sure we find the best way to make the ground work for this in 1.2.6.
Michael and me had a short meeting today talking about the possibilities of the version control. Bug bso#8181 is related to all of this. - Let smbtad maintain an extra table with the list of seen VFS modules and their subversion number, and the number of common data blocks they transfer - Let smbtad store it's own version in this table too - To be fast, let smbtad cache this information in the systems RAM - Let smbtatools version be stored locally, ( in ~./smbtatools-config ), so that they detect when they are first run with a newer version. - Let smbtaquery have a "selfcheck" feature, that looks up the information from smbtad in the database, and creates a report for the user about what has to be done to have the software chain up to date. - Add the selfcheck function to webSMBTA. - Store the latest release version information on morelias, to make it possible to look it up online. - Let smbtaquery also check morelias if newer versions are available upstream.
I am beginning with the smbtad part of this bug. - At startup, store the smbtad version in the table - if a VFS module connection happens, store a flag, and on the very first package we receive, store the module version in the database, plus the IP adress of the sender. - This way, a list of connection doen't even need to be hold in the systems RAM,
commit a0d8630e5c5afea98ecf46933005d4a6b0072b32 Author: Holger Hetterich <hhetter@novell.com> Date: Sun Jul 3 21:47:22 2011 +0200 First commit, tested, of an infrastructure for version control. Additionally, smbtad is storing it's complete configuraten in a table. For the version, smbtad is storing two values in the table, one is the "database version" and another one is the "smbtad version" smbtad will refuse to start if the database version is not the same as the smbtad version. It will then advice the user to run smbtaquery -C, or smbtad -T, and exit. Also a module-list table is being created but this feature is not yet implemented (listing module protocol subversions).
Created attachment 6670 [details] -q 'system, smbta-report full' output A by-product of this bug is the new object "system", and so a query like: -q 'system, smbtad-report full;' generates the attached output. The intention is to have smbtad-report full and smbtad-report short, "full" returns fully documented output for the end-user, like the attachment, and "short" reports just the raw configuration data, to be fetched in bug reports by developers.
Newest commits implementing HTML output for "smbtad-report full"
I have lots of pseudo code in my local branch for smbtad implementing the module table, which I will commit in the upcoming days when it builds. It does require a change of the module table, to make the IP address a unique key.
have it building locally, doing some basic tests and add some of the missing stuff before committing.
f67c64541c2ae1af39ab0e7f98c4fa5d59f636b4 in devel. implements the module table. In case a VFS module connects via a unix domain socket, the table simply says: "unix", otherwise the ip address of the system where the module is running is stored and used as the unique key of the table. This is minimal tested, and anyone interested in trying this is highly welcome. Steps to test are: - create a new empty database - run smbtad -T to set it up (the format has changed so we need to setup a new database) - run traffic - inspect the modules table: select * from modules; For each Samba server that is connecting to smbtad, a modules should be listed, with some base information.
forget to add something to the test description: - make sure to quit the first smb connection (like with smbclient quit, and reconnect) - the module table is being updated per connection, and on the second connect from the same samba server, the table should stay the same.
commit fca1fe796d2491fde1011621d45738fb20815baa shall enable the "smbta-report short" output which is nothing more then a list of configuration parameters in the ascii-stylesheet. HTML to follow.
The new modules table implementation might have a performance impact (although I believe it is minimal as a module is only registered once per SMB connection)
For the notes I've tested in a virtual machine that both smbtad -T and smbtaquery -C work fine with the most recent changes.
Tested in a virtual machine, the modules table works fine so far. I am on a unix socket right now and need to test via IP also.
Summing all that up to a current status. The current code seems to work for IP and unix domain socket connections. What is currently needed is someone testing the following: - configure smbtad for internet socket operation - create an empty database - run smbtad -T - run smbtad for normal operation - create some traffic - post output of "select * from modules;" in this bug In my tests, I get 0.0.0.0 as the module's IP address which is of course not right. But I am not sure if this is caused by the bridged network setup of my virtual machine, as I can see several daemons running connections to 0.0.0.0, not only SMBTA. ( And VirtualBox documents bridged network as only "minimal" implemented ). Therefore I'd like to see someone testing the above steps. If anyone is willing to help, that would be great. I am working on all the other parts of this bug meanwhile.
Hi, I'll do it later today. R
Creating database: NOTICE: CREATE TABLE / UNIQUE will create implicit index "modules_module_ip_address_key" for table "modules" Succesfully created database tables. After performing some traffic: smbtad3=> SELECT * from data; vfs_id | username | usersid | share | domain | timestamp | string1 | length | result | string2 --------+----------+---------+-------+--------+-----------+---------+--------+--------+--------- (0 rows) smbtad3=> SELECT * from modules; module_subrelease_number | module_common_blocks_overflow | module_ip_address --------------------------+-------------------------------+------------------- (0 rows) Only SELECT * from status; shows me some data. Rob
sorry must be too tired to test properly. I didn't spot that all VFS module part was commented out when I tested last time. Right now it looks better. We can still see 0.0.0.0 in IP address: smbtad3=> SELECT * from modules; module_subrelease_number | module_common_blocks_overflow | module_ip_address --------------------------+-------------------------------+------------------- 0 | 1 | 0.0.0.0 (1 row)
The issue of "0.0.0.0" in the module table will be fixed by bug#8321
Uploaded "CURRENT_SMBTATOOLS_VERSION" and "CURRENT_SMBTAD_VERSION" to morelias, they simply contain the current version (for testing this is 1.2.6), and will be used by the self-check function to look up if newer SMBTA versions are available.
Added a libcurl check for smbtaquery, we'll use curl to fetch the version data from morelias.
The recent commmits in smbtatools enable online-checking for updated versions by employing libCURL. A CURL check has been added to the build. So far, the function is able to output in ascii mode : SMBTAD version, SMBTATOOLS version, check the upstream versions and comment on the situation. This is the base of the self-check function, but it will get more detailed in the next step. We shall also check the database-version, so that the user can be told to update or convert the database to a new version, and the self-check function shall print out and comment on the modules tables and their versions.
Example output in case a correct SMBTA installation is set up, but no traffic occured till now (so that smbtad does not have any entries in the module table). holger@linux-lm9n:~/Dev/smbtatools/run> ./smbtaquery -H localhost -P null -S dbuser -N smbta -M pgsql -q 'system, self-check online;' <?xml version="1.0" encoding="UTF-8"?> Your installed smbtad version is 1.2.6. Versions are ok. You have the latest released smbtad version installed. ------------------------------------------------------------------------------ Your installed smbtatools version is 1.2.6. Versions are ok. You have the latest released smbtatools version installed. ------------------------------------------------------------------------------ The version of the database is 1.2.6. Database status is ok. The database is up to date, and matches the installed smbtad version. ------------------------------------------------------------------------------ Seen Samba servers by SMBTA: No servers running VFS modules have been seen by smbtad. Either you have not yet configured any Samba servers to run SMBTA against smbtad, or you did not yet produce any traffic on the mentioned services. Please check your configuration. ------------------------------------------------------------------------------ SMB Traffic Analyzer 1.2.6-devel, 2011-09-06 11:52:24 holger@linux-lm9n:~/Dev/smbtatools/run>
Example output if SMBTA has seen two Samba servers, one already sends more common data, the other is a perfect fit. holger@linux-lm9n:~/Dev/smbtatools/run> ./smbtaquery -H localhost -P null -S dbuser -N smbta -M pgsql -q 'system, self-check online;' <?xml version="1.0" encoding="UTF-8"?> Your installed smbtad version is 1.2.6. Versions are ok. You have the latest released smbtad version installed. ------------------------------------------------------------------------------ Your installed smbtatools version is 1.2.6. Versions are ok. You have the latest released smbtatools version installed. ------------------------------------------------------------------------------ The version of the database is 1.2.6. Database status is ok. The database is up to date, and matches the installed smbtad version. ------------------------------------------------------------------------------ Seen Samba servers by SMBTA: Server IP : 192.168.178.4 Protocol version: 0 Overflow data : 1 Comment: OK - This server does transfer more data than is required by your SMBTA installation. This is fine, as the overflow data will be ignored by smbtad, so it does not harm your installation. The additional data might be a feature that is currently in development and the SMBTA software suite will be updated accordingly at a later point in time. -------------------------------------------------------- Server IP : 192.168.178.5 Protocol version: 0 Overflow data : 0 Comment: OK - This server does transfer exactly the same amountof data that the receiver smbtad requires. It is a perfect fit for your installation. -------------------------------------------------------- ------------------------------------------------------------------------------ SMB Traffic Analyzer 1.2.6-devel, 2011-09-06 12:15:48 holger@linux-lm9n:~/Dev/smbtatools/run>
Example output supposing there is a newer version of SMBTA upstream, done with a real online check and a faked CURRENT_SMBTA_VERSION file set to "1.2.7". holger@linux-lm9n:~/Dev/smbtatools/run> ./smbtaquery -H localhost -P null -S dbuser -N smbta -M pgsql -q 'system, self-check online;' <?xml version="1.0" encoding="UTF-8"?> Your installed smbtad version is 1.2.6. Attention: there is a newer version of smbtad available at http://holger123.wordpress.com/smb-traffic-analyzer/smb-traffic-analyzer-download/ ------------------------------------------------------------------------------ Your installed smbtatools version is 1.2.6. Attention: there is a newer version of smbtatools available at http://holger123.wordpress.com/smb-traffic-analyzer/smb-traffic-analyzer-download/ ------------------------------------------------------------------------------ The version of the database is 1.2.6. Database status is ok. The database is up to date, and matches the installed smbtad version. ------------------------------------------------------------------------------ Seen Samba servers by SMBTA: Server IP : 192.168.178.4 Protocol version: 0 Overflow data : 1 Comment: OK - This server does transfer more data than is required by your SMBTA installation. This is fine, as the overflow data will be ignored by smbtad, so it does not harm your installation. The additional data might be a feature that is currently in development and the SMBTA software suite will be updated accordingly at a later point in time. -------------------------------------------------------- Server IP : 192.168.178.5 Protocol version: 0 Overflow data : 0 Comment: OK - This server does transfer exactly the same amountof data that the receiver smbtad requires. It is a perfect fit for your installation. -------------------------------------------------------- ------------------------------------------------------------------------------ SMB Traffic Analyzer 1.2.6-devel, 2011-09-06 12:20:03 holger@linux-lm9n:~/Dev/smbtatools/run>
I now continue to work on the HTML stylesheets for this
HTML output is done. Benjamin, this function can now be integrated into webSMBTA. It should be only available in the administrator role.
I am trying to fixup the problems in smbta-report, and work on a "configuration file" stylesheet for it.
Basic implementation for websmbta is done. Reassigning to Holger, because of the stylesheet issues and the config-file output.
I've checked the web output of this function and everything should be implemented now. Closing this.