Bug 8206 - Integrated version control
Integrated version control
Status: RESOLVED FIXED
Product: smbta
Classification: Unclassified
Component: general
unspecified
All All
: P5 normal
: ---
Assigned To: Holger Hetterich
:
Depends on: 8181 8321
Blocks: 8204 8285
  Show dependency treegraph
 
Reported: 2011-06-04 21:42 UTC by Holger Hetterich
Modified: 2012-04-25 18:09 UTC (History)
5 users (show)

See Also:


Attachments
-q 'system, smbta-report full' output (3.79 KB, text/plain)
2011-07-07 18:36 UTC, Holger Hetterich
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Holger Hetterich 2011-06-04 21:42:04 UTC
This will be the second important feature implementation for 1.2.6. SMBTA shall have integrated version control for all components.

Some facts about this:
-> smbtad can log the VFS modules sub-version number in the database
-> smbtad can log it's own version information in the database
-> smbtad can log the fact that it is ignoring new common data blocks in the database
-> smbtatools can locally store their version information and react on update
-> smbtatools can lookup what smbtad is storing
-> smbtatools can proactively run the database conversion process if that is required
-> all that can be controlled from webSMBTA or smbtaquery

I have some thoughts for this. Benjamin, Michael, let us discuss this in a meeting to get the overview, and create a mockup of the different version options. Robert, if you are also interested in this, please say so, we'll find ways, IRC or voIP. I am sure we find the best way to make the ground work for this in 1.2.6.
Comment 1 Holger Hetterich 2011-06-07 10:44:25 UTC
Michael and me had a short meeting today talking about the possibilities of the version control.

Bug bso#8181 is related to all of this.

- Let smbtad maintain an extra table with the list of seen VFS modules and their subversion number, and the number of common data blocks they transfer
- Let smbtad store it's own version in this table too
- To be fast, let smbtad cache this information in the systems RAM


- Let smbtatools version be stored locally, ( in ~./smbtatools-config ), so that they detect when they are first run with a newer version.
- Let smbtaquery have a "selfcheck" feature, that looks up the information from smbtad in the database, and creates a report for the user about what has to be done to have the software chain up to date.
- Add the selfcheck function to webSMBTA.
- Store the latest release version information on morelias, to make it possible to look it up online.
- Let smbtaquery also check morelias if newer versions are available upstream.
Comment 2 Holger Hetterich 2011-07-02 15:12:44 UTC
I am beginning with the smbtad part of this bug.

- At startup, store the smbtad version in the table
- if a VFS module connection happens, store a flag,
  and on the very first package we receive, store the
  module version in the database, plus the IP adress of
  the sender.
- This way, a list of connection doen't even need to be hold
  in the systems RAM,
Comment 3 Holger Hetterich 2011-07-03 19:55:02 UTC
commit a0d8630e5c5afea98ecf46933005d4a6b0072b32
Author: Holger Hetterich <hhetter@novell.com>
Date:   Sun Jul 3 21:47:22 2011 +0200

    First commit, tested, of an infrastructure for version control.
    Additionally, smbtad is storing it's complete configuraten in a
    table.
    
    For the version, smbtad is storing two values in the table, one
    is the
    "database version"
    and another one is the
    "smbtad version"
    
    smbtad will refuse to start if the database version is not the
    same as the smbtad version. It will then advice the user to
    run smbtaquery -C, or smbtad -T,  and exit.
    
    Also a module-list table is being created but this feature is not
    yet implemented (listing module protocol subversions).
Comment 4 Holger Hetterich 2011-07-07 18:36:52 UTC
Created attachment 6670 [details]
-q 'system, smbta-report full' output

A by-product of this bug is the new object "system", and so a query like:
-q 'system, smbtad-report full;'

generates the attached output. The intention is to have smbtad-report full and smbtad-report short, "full" returns fully documented output for the end-user, like the attachment, and "short" reports just the raw configuration data, to be fetched in bug reports by developers.
Comment 5 Holger Hetterich 2011-07-12 21:48:11 UTC
Newest commits implementing HTML output for "smbtad-report full"
Comment 6 Holger Hetterich 2011-07-18 20:00:35 UTC
I have lots of pseudo code in my local branch for smbtad implementing the module table, which I will commit in the upcoming days when it builds.

It does require a change of the module table, to make the IP address a unique key.
Comment 7 Holger Hetterich 2011-07-19 15:23:58 UTC
have it building locally, doing some basic tests and add some of the missing stuff before committing.
Comment 8 Holger Hetterich 2011-07-19 20:36:04 UTC
f67c64541c2ae1af39ab0e7f98c4fa5d59f636b4 in devel.

implements the module table. In case a VFS module connects via a unix domain socket, the table simply says: "unix", otherwise the ip address of the system where the module is running is stored and used as the unique key of the table.

This is minimal tested, and anyone interested in trying this is highly welcome. Steps to test are:

- create a new empty database
- run smbtad -T to set it up (the format has changed so we need to setup a new database)
- run traffic
- inspect the modules table:
  select * from modules;

For each Samba server that is connecting to smbtad, a modules should be listed, with some base information.
Comment 9 Holger Hetterich 2011-07-19 20:45:13 UTC
forget to add something to the test description:

- make sure to quit the first smb connection (like with smbclient quit, and reconnect)
- the module table is being updated per connection, and on the second connect from the same samba server, the table should stay the same.
Comment 10 Holger Hetterich 2011-07-19 21:21:24 UTC
commit fca1fe796d2491fde1011621d45738fb20815baa shall enable
the "smbta-report short" output which is nothing more then a list of configuration parameters in the ascii-stylesheet.

HTML to follow.
Comment 11 Holger Hetterich 2011-07-19 21:23:26 UTC
The new modules table implementation might have a performance impact (although I believe it is minimal as a module is only registered once per SMB connection)
Comment 12 Holger Hetterich 2011-07-22 11:43:25 UTC
For the notes I've tested in a virtual machine that both smbtad -T and smbtaquery -C work fine with the most recent changes.
Comment 13 Holger Hetterich 2011-07-22 11:59:09 UTC
Tested in a virtual machine, the modules table works fine so far. I am on a unix socket right now and need to test via IP also.
Comment 14 Holger Hetterich 2011-07-24 18:10:00 UTC
Summing all that up to a current status.

The current code seems to work for IP and unix domain socket connections.

What is currently needed is someone testing the following:
- configure smbtad for internet socket operation
- create an empty database
- run smbtad -T
- run smbtad for normal operation
- create some traffic
- post output of "select * from modules;" in this bug

In my tests, I get 0.0.0.0 as the module's IP address which is of course not right. But I am not sure if this is caused by the bridged network setup of my virtual machine, as I can see several daemons running connections to 0.0.0.0, not only SMBTA. ( And VirtualBox documents bridged network as only "minimal" implemented ).

Therefore I'd like to see someone testing the above steps. If anyone is willing to help, that would be great.

I am working on all the other parts of this bug meanwhile.
Comment 15 Robert Piasek 2011-07-25 08:13:21 UTC
Hi,

I'll do it later today.

R
Comment 16 Robert Piasek 2011-07-26 14:17:36 UTC
Creating database:

NOTICE:  CREATE TABLE / UNIQUE will create implicit index "modules_module_ip_address_key" for table "modules"

Succesfully created database tables.


After performing some traffic:

smbtad3=> SELECT * from data;
 vfs_id | username | usersid | share | domain | timestamp | string1 | length | result | string2 
--------+----------+---------+-------+--------+-----------+---------+--------+--------+---------
(0 rows)

smbtad3=> SELECT * from modules;
 module_subrelease_number | module_common_blocks_overflow | module_ip_address 
--------------------------+-------------------------------+-------------------
(0 rows)

Only SELECT * from status; shows me some data.


Rob
Comment 17 Robert Piasek 2011-07-26 15:28:54 UTC
sorry must be too tired to test properly. I didn't spot that all VFS module part was commented out when I tested last time.

Right now it looks better. We can still see 0.0.0.0 in IP address:

smbtad3=> SELECT * from modules;
 module_subrelease_number | module_common_blocks_overflow | module_ip_address 
--------------------------+-------------------------------+-------------------
                        0 |                             1 | 0.0.0.0
(1 row)
Comment 18 Holger Hetterich 2011-07-26 20:29:59 UTC
The issue of "0.0.0.0" in the module table will be fixed by bug#8321
Comment 19 Holger Hetterich 2011-07-27 19:47:07 UTC
Uploaded "CURRENT_SMBTATOOLS_VERSION" and "CURRENT_SMBTAD_VERSION" to morelias, they simply contain the current version (for testing this is 1.2.6), and will be used by the self-check function to look up if newer SMBTA versions are available.
Comment 20 Holger Hetterich 2011-07-27 20:21:53 UTC
Added a libcurl check for smbtaquery, we'll use curl to fetch the version data from morelias.
Comment 21 Holger Hetterich 2011-08-31 16:13:21 UTC
The recent commmits in smbtatools enable online-checking for updated versions by employing libCURL. A CURL check has been added to the build. So far, the function is able to output in ascii mode : SMBTAD version, SMBTATOOLS version, check the upstream versions and comment on the situation.

This is the base of the self-check function, but it will get more detailed in the next step. We shall also check the database-version, so that the user can be told to update or convert the database to a new version, and the self-check function shall print out and comment on the modules tables and their versions.
Comment 22 Holger Hetterich 2011-09-06 09:56:09 UTC
Example output in case a correct SMBTA installation is set up, but no traffic occured till now (so that smbtad does not have any entries in the module table).


holger@linux-lm9n:~/Dev/smbtatools/run> ./smbtaquery -H localhost -P null -S dbuser -N smbta -M pgsql -q 'system, self-check online;'
<?xml version="1.0" encoding="UTF-8"?>



Your installed smbtad version is 1.2.6.

Versions are ok. You have the latest released smbtad version installed.
------------------------------------------------------------------------------

Your installed smbtatools version is 1.2.6.

Versions are ok. You have the latest released smbtatools version installed.
------------------------------------------------------------------------------

The version of the database is 1.2.6.

Database status is ok. The database is up to date, and matches the installed smbtad version.
------------------------------------------------------------------------------

Seen Samba servers by SMBTA:


No servers running VFS modules have been seen by smbtad. Either you have not yet configured any Samba servers to run SMBTA against smbtad, or you did not yet produce any traffic on the mentioned services. Please check your configuration.

------------------------------------------------------------------------------
SMB Traffic Analyzer 1.2.6-devel, 2011-09-06 11:52:24

holger@linux-lm9n:~/Dev/smbtatools/run>
Comment 23 Holger Hetterich 2011-09-06 10:16:54 UTC
Example output if SMBTA has seen two Samba servers, one already sends more common data, the other is a perfect fit.


holger@linux-lm9n:~/Dev/smbtatools/run> ./smbtaquery -H localhost -P null -S dbuser -N smbta -M pgsql -q 'system, self-check online;'
<?xml version="1.0" encoding="UTF-8"?>



Your installed smbtad version is 1.2.6.

Versions are ok. You have the latest released smbtad version installed.
------------------------------------------------------------------------------

Your installed smbtatools version is 1.2.6.

Versions are ok. You have the latest released smbtatools version installed.
------------------------------------------------------------------------------

The version of the database is 1.2.6.

Database status is ok. The database is up to date, and matches the installed smbtad version.
------------------------------------------------------------------------------

Seen Samba servers by SMBTA:



Server IP	: 192.168.178.4
Protocol version: 0
Overflow data	: 1
Comment: OK - This server does transfer more data than is required by your SMBTA installation. This is fine, as the overflow data will be ignored by smbtad, so it does not harm your installation. The additional data might be a feature that is currently in development and the SMBTA software suite will be updated accordingly at a later point in time.
--------------------------------------------------------



Server IP	: 192.168.178.5
Protocol version: 0
Overflow data	: 0
Comment: OK - This server does transfer exactly the same amountof data that the receiver smbtad requires. It is a perfect fit for your installation. 
--------------------------------------------------------




------------------------------------------------------------------------------
SMB Traffic Analyzer 1.2.6-devel, 2011-09-06 12:15:48

holger@linux-lm9n:~/Dev/smbtatools/run>
Comment 24 Holger Hetterich 2011-09-06 10:20:55 UTC
Example output supposing there is a newer version of SMBTA upstream, done with a real online check and a faked CURRENT_SMBTA_VERSION file set to "1.2.7".


holger@linux-lm9n:~/Dev/smbtatools/run> ./smbtaquery -H localhost -P null -S dbuser -N smbta -M pgsql -q 'system, self-check online;'
<?xml version="1.0" encoding="UTF-8"?>



Your installed smbtad version is 1.2.6.

Attention: there is a newer version of smbtad available at http://holger123.wordpress.com/smb-traffic-analyzer/smb-traffic-analyzer-download/
------------------------------------------------------------------------------

Your installed smbtatools version is 1.2.6.

Attention: there is a newer version of smbtatools available at http://holger123.wordpress.com/smb-traffic-analyzer/smb-traffic-analyzer-download/
------------------------------------------------------------------------------

The version of the database is 1.2.6.

Database status is ok. The database is up to date, and matches the installed smbtad version.
------------------------------------------------------------------------------

Seen Samba servers by SMBTA:



Server IP	: 192.168.178.4
Protocol version: 0
Overflow data	: 1
Comment: OK - This server does transfer more data than is required by your SMBTA installation. This is fine, as the overflow data will be ignored by smbtad, so it does not harm your installation. The additional data might be a feature that is currently in development and the SMBTA software suite will be updated accordingly at a later point in time.
--------------------------------------------------------



Server IP	: 192.168.178.5
Protocol version: 0
Overflow data	: 0
Comment: OK - This server does transfer exactly the same amountof data that the receiver smbtad requires. It is a perfect fit for your installation. 
--------------------------------------------------------




------------------------------------------------------------------------------
SMB Traffic Analyzer 1.2.6-devel, 2011-09-06 12:20:03

holger@linux-lm9n:~/Dev/smbtatools/run>
Comment 25 Holger Hetterich 2011-09-06 10:25:25 UTC
I now continue to work on the HTML stylesheets for this
Comment 26 Holger Hetterich 2011-09-06 12:11:44 UTC
HTML output is done. Benjamin, this function can now be integrated into webSMBTA. It should be only available in the administrator role.
Comment 27 Holger Hetterich 2011-09-06 12:22:02 UTC
I am trying to fixup the problems in smbta-report, and work on a "configuration file" stylesheet for it.
Comment 28 Benjamin Brunner 2011-09-06 18:11:36 UTC
Basic implementation for websmbta is done.
Reassigning to Holger, because of the stylesheet issues and the config-file output.
Comment 29 Holger Hetterich 2012-04-25 18:09:05 UTC
I've checked the web output of this function and everything should be implemented now. Closing this.