TMVA News Archive

TMVA version 4.2.0 is included in ROOT release 5.34/11.
The changes with respect to TMVA-v4.1.2 / ROOT 5.30 are listed below.

Factory:

• NormMode: as called in PrepareTrainingAndTestTree
  The default has changed and is now "EqualNumEvents" (with fixed meaning) While previously NumEvents and EqualNumEvents (by mistake/miscommunication) took into account training+test events, they are now correctly normalising only "Training Events" (note the reason for these normalisations was to have the possibility to easily force the effective (weighted) number of training events used for Signal (class 0) to equal the number of training events in the Backgr. (sum of all remaining classes in multiclass mode)

  NumEvents:

  - the weighted number of events is scaled, independently for signal and backgroundm, such that the sum of weights equals the number of events given in the Factory::PrepareTrainingAndTestTree("",nTrain_Signal=3000,nTrain_Background=6000) call. This example call will give hence end up in having 2x more background events in the training compared to the signal, no matter what the individual event weights have been. (watch out! if you specify nTrain_Signal=0,nTrain_Background=0), then the ratio will be according to total numbers of MC events in the signal and background respectively, which could be very different from the usually good ratio of having about the same weighted number signal and background events in the training. In that case it is better to use:

  EqualNumEvents:

  - for the signal events, the same is done as for NumEvents. The background events however are reweighted such, that their sum of weights equals that for the signal events. This results in the same effective (weighted) number of signal and background events to be seen in the training.

• Transformations=I is default again in Factory (this defines which variables distribution plots are added to the TMVA output file - and hence displayable via the TMVAGui)

Boosted Decision Trees:

• Some changes to the training options:

nEventsMin (deprecated) please replace by → MinNodeSize

The option nEventsMin which specified the minimum number of training event in a leaf node as an absolute number has been replaced by "MinNodeSize" which is given in "percentage of the trainin sample". Like this the training options become less dependent on the actual number of training sample size

NNodesMax (deprecated) please replace by → MaxDepth

GradBaggingFraction and UseNTrainEvents replaced by BaggedSampleFraction

- they both meant the same thing and are now deprecated → use BaggedSampleFraction instead

UseWeightedTrees → removed

- it was default anyway and the only reasonable choice there is

PruneBeforeBoost → removed

- it has been mostly a debug/trial option

NegWeightTreatment=IgnoreNegWeights → replaced by NegWeightTreatment=IgnoreNegWeightsInTraining

- Unfortunatly the default "IgnoreNegWeights" to the BDT option "NegWeightTreatment" collided with the a global option and had to be replaced.

MethodBoost:

• some cleanup (removed strange experimental boosting option HighEdgeGaus, HighEdgeCoPara ..... )
• remove options MethodWeightType... have it defined by the Boost Method (these have been trial options.. but for clarity it is much better to stick to the "standard" ones (i.e log(alpha) for AdaBoost etc)
• up to now, the first classifier was trained with the full sample, I think however, it should also be a bagged sample (i.e. particularily if smaller sample sizes for the bagged samples were demanded) .. it's changed now, accordingly

TMVA version 4.1.2 is included in ROOT release 5.30/00.
The changes with respect to TMVA-v4.1.0 / ROOT 5.28 are listed below.

Variable transformation:

• Variable transformations can now be applied to a user-defined subset of variables (and regression targets).
• Variable transformations are now also possible for for generalised boosting.
• Extended PDEFoam functionality:
  
  • Multiclass classification by training of one discriminator foam for each variable.
  • The cell tree can now be plotted from the macro test/PlotFoams. This simplifies the comparison of the PDEFoam structure with a decision tree.
  • Variable importance ranking by counting the number of cuts made in each dimension. The variable with which the maximum number of cuts was performed is ranked highest.
• Fixed the size of the sampling box in PDEFoam: In TMVA-v4.1.0 the size of the PDEFoam sampling box in each dimension was 2*VolFrac times the foam size. This was contrary to the intention and the documentation in the UserGuide and is now corrected: in TMVA-v4.1.2 the size of the PDEFoam sampling box in each dimension is now VolFrac times the foam size. This implies that in TMVA-v4.1.2 the VolFrac value for training a PDEFoam must be doubled in order to give the same results as in TMVA-v4.1.0. The default VolFrac value was also changed from 0.0333 to 0.0666.
• New configuration variable "NbinsMVAoutput" defining the bins of the MVA output variables in the TMVA training plots produced via the GUI. As always, Config settings can be modified in the training script via, eg, the command
  (TMVA::gConfig().GetVariablePlotting()).fNbinsMVAoutput = 50;
  to be called after initialising the TMVA Factory object.

Bug fixes:

• Requested number of training and testing events was not correct when pre-selection cuts were applied. Now the number of requested events scales with the preselection efficiency and hence does not need to be adjusted with the pre-selection. This also corrects the problems seen in the Category classifier, where pre-selection is used to build the categories.
• Correct histogram boundaries in PlotVariable.
• Correct scanning procedure in OptimizeTuningParameters.
• Print the significance formula that is actually used
• Small speed improvement for PDEFoam functions.
• Fix for MethodBoost which ensures that the method options for the boosted classifier are handled correctly during boosting.
• Fixed problems in classification of some methods when booking background training tree before signal one.
• Fixed preprocessing transformation bug in HMatrix.
• Several minor bug fixes for version 4.1.2.

The most important new feature is the support for the simulataneous classification of multiple output classes for several multivariate methods. A lot of effort went into consolidation of the software, i.e. method performance and robustness, and framework stability. TMVA 4.1.0 is included in the ROOT production release 5.28.

The changes with respect to TMVA 4.0.7 are listed below.

Framework:


• Multi-class support. Support of multiple output classes (i.e., more than a single background and signal class) has been enabled for the methods: MLP (NN), BDTG, FDA. The multiclass functionality can be enabled with the Factory option "AnalysisType=multiclass". Training data are specified with an additional classname: factory->AddTree(tree,"myclassname");. After the training a genetic algorithm is invoked to determine the best cuts for selecting a specific class, based on the figure of merit: purity*efficiency. TMVA comes with two examples for the use of mutiple output classes in $ROOTSYS/tmva/test: TMVAMulticlass.C and TMVAMulticlassApplication.C

• New TMVA event vector building. The code for splitting the input data into training and test samples for all classes and the mixing of those samples to one training and one test sample has been rewritten. The new code performs better and has a clearer structure. This fixes several bugs reported by TMVA users.

• Code and performance test framework. A unit test framework for nightly software and method performance validation has been implemented.

Methods:

• BDT Automatic parameter optimisation for building the tree architecture. The optimisation procedure uses the performance of the trained classifier on the "test sample" for finding the set of optimal parameters. Two different optimisation methods are available: parameger scanning and genetic algorithm. Currently, parameter optimisation is implemented only for the three parameters that influence the tree architecture: the maximum depth of a tree, MaxDepth, the minimum number of events in each node, NodeMinEvents, and the number of tress, NTrees. Optimisation is invoked by calling factory->OptimizeAllMethods(), prior to the call factory->TrainAllMethods(). Automated and configurable parameter optimisation is soon to be enabled for all methods via the central framework (for those parameters where optimisation is applicable).

• BDT node splitting. While Decision Trees typically have only univariate splits, in TMVA one can now also opt for multivariate splits that use a "Fisher Discriminant" (option: UseFisherCuts), built from all observables that show correlations larger than some settable threshold (MinLinCorrForFisher). The training will then test at each split a cut on this Fisher discriminant in addition to all univariate cuts on the variables (or only on those variables that have not been used in the Fisher discriminant, option UseExcusiveVars). No obvious improvement betwen very simple decision trees after boosting has been observed so far, but only a limited number of studies has been performed concerning potiential benenfit of these simple multivariate splits.

Bug fixes:

• A problem in the BDTG has been fixed, leading to a much improved regression performance.
• A problem in the TMVA::Reader has been fixed.
• With the new test framework and the coverity checks of ROOT a number of bugs were discovered and fixed. They mainly concerned memory leaks, and did not affect the performance.

• New Category method allowing the user to separate the training data (and accordingly the application data) into disjoint sub-populations exhibiting significantly different properties. The separation into phase space regions is done by applying requirements on the input and/or spectator variables. In each of these disjoint regions (each event must belong to one and only one region), an independent training is performed using the most appropriate MVA method, training options and set of training variables in that zone. The division into categories in presence of distinct sub-populations reduces the correlations between the training variables, improves the modelling, and hence increases the classification and regression performance. Presently, the Category method works for classification only, but regression will follow soon. Please contact us if urgently needed.
• Added new example scripts and data files illustrating how the new Category method is configured and used. Please check:

  TMVA/macros/ TMVAClassificationCategory.C	TMVA/macros/ TMVAClassificationCategoryApplication.C
  TMVA/execs/ TMVAClassificationCategory.cxx	TMVA/execs/ TMVAClassificationCategoryApplication.cxx

• The Users Guide has been updated.
• Added regression functionality for gradient boosted trees using a Huber loss function.
• The analysis type is no longer defined by calling a dedicated TestAllMethods-member-function of the Factory, but with the option "AnalysisType" in the Factory. The default value is "Auto" where TMVA tries to determine the most suitable analysis type from the targets and classes the user has defined. Other values are "regression", "classification" and "multiclass" for the forthcoming multiclass classification.
• Removed obsolete option "FVerySmart" from Cuts method.
• Bug fixes:

  • Spectators and Targets could not be used with by-hand assignment of events.
  • Corrected types (training/testing) for assigning single events.
  • Changed message from FATAL to WARNING when the user requests more events for training or testing than available.
  • Fixed bug which caused TMVA to crash if the number of input variables exceeded the allowed maximum for generating scatter plots.
  • Prevent TMVA from crashing when running with an empty TTree or TChain.
  • Added missing regression evaluation plots for training sample.

• Checks are performed if events are unvoluntarily cut by using a non-filled array entry (e.g. "arr[4]" is used, when the array has not always at least 5 entries). A warning is given in that case.
• Display of convergence information in the progress bar for MLP during training.
• Creation of animated gifs for MLP convergence monitoring (please contact authors if you want to do this for your MLP).

• Variables, targets and spectators are now checked if they are constant. The execution of TMVA is stopped for variables and targets, a warning is given for spectators.
• Bug fixes:

  • A variable expression like "Alt$(arr[3],0)" can now be used to give a default value for a variable if for some events the array don't contain enough elements (e.g. in two jet events, sometimes only one jet is found and thus, the array jetPt[] has only one entry in that cases).
  • Plot ranges for scatter-plots showing the transformed events are now correct.
  • User defined training/testing-trees are now handled correctly.
  • Fix bug in correlation computation for regression
  • Consistent use of variable labels (for the log output) and variable titles (in histograms)

• "Spectator" variables can be defined now which are computed just as the input variables and which are written out into the TestTree, but which don't participate in any MVA calculation (useful for correlation studies).
• New booking option "IgnoreNegWeightsInTraining" to test the effect of events with negative weights on the training. This is especially useful for methods, which do not properly deal with such events.
• Bug fixes:

  • Fixed regression bug in VariableNormalizeTransform (use number of targets from Event instead of DataSet).
  • Fixed Multitarget-Regression in PDEFoam, foam dimensions were miscalculated.
  • Added writing of targets to the weight files in regression mode to fix problems in RegressionApplication.
  • Added missing standard C++ header files missing to some classes, which lead to compilation failures on some architectures (thanks to Lucian Ancu, Nijmegen, for reporting these).
  • Added checks for unused options to Factory and DataSetFactory configuration options interpretation. Will now complain if wrong option labels are used.
  • Fixed standard creation of correlation matrix plots.
  • Fixed internal mapping problem giving a fatal error message when destroying and recreating the Factory.
  • A number of non-intrusive modifications required for integration into ROOT v5.23.

Release of beta version of TMVA 4
Why a beta release ?	The changes from TMVA 3.9.7 to TMVA 4 are substantial. Some parts such as documentation − the TMVA Users Guide − are not yet updated accordingly (this is our next target!). We have done a large number of tests before releasing this version of TMVA, and are thus confident that the classifiers work properly. For the regression algorithms we need to collect more practical experience to know how best to tune them. Technically however they work. Please provide feedback whenever you encounter a problem or unwanted feature!
What risk do I take by using TMVA 4 ?	TMVA 4 is not (necessarily) backward compatible. Mainly, because we have moved from text weight files to XML format. It is still possible to create MVA methods via the Reader by giving text weight files, but we cannot guarantee that they will work properly due to the many changes we have done in the data handling framework and MVA methods themselves. If you need to upgrade to TMVA 4, but also require backward compatibility for the reading of the weight files, contact the authors − we will help.
Known problems / issues	We have not yet tested the Plugin-Handler with TMVA 4 Some of the specific RuleFit performance analysis macros do not work yet The TMVA Users Guide is not yet updated

---

• Summary of main new features:
• Reorganisation of internal data handling and constructors of methods to allow to build arbitrary composite MVA methods, and to deal with multi-class classification and multi-target regression
• Extended TMVA to multivariate multi-target regression
• Any TMVA method can now be boosted (linearly or non-linearly)
• Transformation of input variables can be chained as wished
• Weight files are now in XML format
• New MVA methods "PDE-Foam" and "LD", both featuring classification and regression
• Moved from CVS to SVN concurrent development system
Comments:
• On XML format:
The old text format is obsolete though still readable in the application. Backward compatibility is NOT guaranteed. Please contact the authors if you require the reading of old text weight files in TMVA 4.
• Standard macros:
The structure of the standard macros has changed: macros are still in the "TMVA/macros" directory, but distringuished for classification and regression examples:
In macros:
TMVAClassification.C, TMVAClassificationApplication.C
TMVARegression.C, TMVARegressionApplication.C
In execs:
TMVAClassification.cxx, TMVAClassificationApplication.cxx
TMVARegression.cx, TMVARegressionApplication.cxx ("execs" replaces the previous "examples", for exectutables)
Classification and regression analysis (training) is analysed as usual via standard macros that can be called from dedicated GUIs (as before).

• Regression:
Not yet available for all MVA methods. It exists for: PDE-RS, PDE-Foam, K-NN, LD, FDA, MLP, BDT for single targets (1D), and MLP for multiple targets (nD).

Not all transformation of input variables are available (only "Norm" so far).

Regression requires specific evaluation tools:
• During the trainin we provide a ranking of input variables, using various criteria: correlations, transposed correlation, correlation ratio, and "mutual information" between input variables and regression target. (Correlation ratio and mutual information implmentations provided by Moritz Backes, Geneva U)
• After the training, the trained MVA methods are ranked wrt. the deviations between regression target and estimate.
• Macros plot various deviation and correlation quantities. A new GUI (macros/TMVARegGui.C) collects these macros.
• Improvements of and new features for MVA methods:
• Linear Discriminant (LD)
Re-implementation of "Fisher" method as general linear discriminant ("LD"), which is also regression capable (so far: single-target only)
• PDEFoam
PDE-Foam is a variation of the PDE-RS method using a self-adapting binning method to divide the multi-dimensional variable space into a finite number of hyper-rectangles (cells). The binning algorithm adjusts the size and position of a predefined number of cells such that the variance of the signal and background densities inside the cells reaches a minimum.
• BDT
Introduced gradient boosting and stochastic gradient boosting for classification with BDT (as desribed by Friedman 1999). See "BDTG" example in TMVAClassification.C/cxx.

Introduced gradient boosting and stochastic gradient boosting for classification with BDT (as desribed by Friedman 1999). See "BDTG" example in TMVAClassification.C/cxx.

A new option allows to restrict the maximum tree depth. This may be used to avoid overtraining and often gives better performance than pruning. (The pruning mechanism needs to be revisited)
• MLP
Introduced recognition of convergence via general ConvergenceTest-class for interrupting computations when convergence is reached. This feature has is used now in MethodMLP. Improved treatment of event-weights in BFGS training.

Implemented random and importance sampling of events in DataSet. Implemented the usage of this feature for MLP.
• TMlpANN (interface to TMultiLayerPerceptron)
  Now also uses event weights and writes standalone C++ class.

• k Nearest Neighbours (k-NN)
A new global knn search function has been added to NodekNN that searches for k-nearest neighbor using event weights instead of raw event counts. ModulekNN has been modified to allow searches using "weight" or "count" option, where "count" is default. Added UseWeight option to MethodKNN to allow using of "weight" or "count". (Work by Rustem Ospanov, CERN).
• PDF class (for Likelihood and MVA outputs)
Adaptive smoothing the PDF class, allowing it to smooth between MinSmoothNum (for regions with more signal) and MaxSmoothNum (for regions with less signal).

Configuration of the PDF parameters from the option string moved to PDF class, allowing the user to define all the PDF functionalities in every classifier the PDF is used (i.e., also for the MVA PDFs). The reading of these variables was removed from MethodBase and MethodLikelihood. This also allows improved (full) PDF configuration of MVA output via the CreateMvaPdf option. (Work by Or Cohen, CERN & Weizmann)
• New generalisation methods:
• MethodCompositeBase: combines more than one classifier within one.
• MethodBoost: boosts/bags any classifier type. A special booking procedure for it was added to Factory class.
• MethodDT: a classifier composed of a single decision tree, boosted using MethodBoost. Results are compatible with BDT, but BDT remains the default for boosted decision trees, because it has pruning (among other additional features).
• Other developments:

• Improved handling of small Likelihood values such that Likelihood performance increases in analyses with many variables (~>10). Thanks to Ralph Schaefer (Bonn U.) for reporting this.

• Nicer plotting: custom variable titles and units can be assigned in "AddVariable" call.
• Introduced the inverse transformation InverseTransform for the variable transformations into the framework. While this is not necessary for classification, it is necessary for regression. The inverse transformation of the normalization transformation has been implemented.
• Started to extend the variable transformations to the regression targets as well.
• MethodCuts now produces the 'optimal-cut' histograms needed by macro mvaeffs.C. (macro 5a of TMVAGui.C)
• MsgLogger can be silenced in order to prevent excess output during boosting.
• Third dataset type added centrally (Training, Validation and Testing). The validation data is split off the original training data set.
• Update of GUI and other Macros according to the new features of PDF and the addition of MethodBoost.

Dec 2, 2008 (TMVA-v3.9.6)

• Added variable binning and interpolation of the cumulative distributions to Gauss-Decorrelation preprocessing (thanks to Diego Martinez). Include proper event-weight treatment.
• BDT: when "bagging" do not count events with weight "0" for weighted and unweighted events. New bagging option "SampleSizeFraction" which gives the size of the resampled event sample relative to the original sample.
• Bug fixes:

• Inserted missing variable initialisations in TSynapse of MLP. Thanks to Jiahang Zhong (Academia Sinica) for finding this.
• Corrected screen print of decorrelation transformation (missing '-' sign in first variable).
• Corrected event-weight treatment in MVA-PDF creation, used for Rarity calculation. Thanks to Bertrand Chapleau (McGill U.) for reporting this. A similar weight bug was fixed in MethodLikelihood.
• Fix that now the number of variables set by "UseNvars" for RanomiseForests is indeed properly set. Before it was typically some value less than the one that was asked for.
• Introduce missing event-weight treatment to Decorrelation preprocessing.
• Corrected mismatch in binning between output and weight file for Cuts classifier (thanks to Bjoern Gosdzik (DESY) for reporting this).
• The macros TMVAnalysis.C and TMVApplication.C now compile in ROOT (via .L TMVAnalysis.C++).

• New Decision Tree Pruning algorithm (Cost Complexity Pruning a la CART) written by Doug Schouten (Fraser U.). It replaces the old CostComplexity and CostComplexity2 algorithms.
• Further improvements introduced by Doug:

• New no splitting option (choosable with NCuts<0) that finds best split point by first sorting the events for each variable and then looping through all events, placing the cuts always in the middle between two of the sorted events, and finding the true possible maximum separation gain in the training sample by cutting on this variable.
• The beta parameter in the AdaBoost is now an option (default is 1).
• The node purity at which a node is classified as signal (respective background node) for determining the error fraction in the pruning became a parameter that can be set via the option NodePurityLimit (default is 0.5).
• First implementation of a new preprocessing method: transformation of the variables first into a Gaussian distribution, then performing a decorrelation of the "Gaussianised" variables. The transformation is again done by default such that (by default) the signal distributions become Gaussian and are decorrelated. Note that simultaneous Gaussianisation and decorrelation of signal and background is only possible (and done) for methods, such as Likelihood, which test both hypotheses.
• Bug fixes:

• Corrected compilation failure on some architectures with ROOT 5.18.00.
• Fix in Expected error pruning: rather than multiplying both sides, the error on the node and the sub-tree, with the prune strength, now only the expected error of the sub-tree is scaled.
• Fix in FDA parsing of the input formula. There were problems when treating more than 10 parameters (thanks to Hugh Skottowe for reporting this).

• New reference page for configuration options. The page is automatically generated for each new release. Next to the classifiers also exist information links for hints to improve the classifier performance (click on the "i" button). Many thanks to Zhiyi Liu (Fraser U) for suggesting this.
• Bug fixes:

• Calculation of "Separation": fixed bin-shift and normalisation bugs. Thanks to Dag Gillberg (Fraser U) for spotting these.
  

• Fixed problem in "SetSignal(Background)WeightExpression": signal (background weight expressions not existing in the background (signal) tree led to an abort of the tree reading ("Bad numerical expression"). Thanks to Alfio Rizzo (Brussels) for pointing this out.
  

• Fixed problem when specifying train and test tree explicitly. Some code was forgotten in the background part, creating incompatibilities. Thanks to Zhiyi Liu (Fraser U) for reporting this.

• Bug fixes:

• Fixed problem introduced in 3.9.0 and 3.9.1: preprocessing cuts could not be applied on a variable that was not declared via AddVariable (thanks to Daniel Stricker, Karlsruhe U., for spotting this).
  

• Corrected configurable random seed in GeneticAlgorithm (thanks to David Gonzalez Maline, CERN, for pointing this out).
  

• Fixed Cuts (optimisaton) method -> event with smallest value was not included in search for optimal cut (thanks to Dimitris Varouchas, LAL-Orsay, for helping us detecting the problem).

• Bug fix:

• Added missing link to libMLP shared library to Makefile; also fixed compilation issue with ROOT 5.08.

• New Simulated Annealing (SA) algorithm for global minimisation in presence of local minima (optionally used in cut optimisation (MethodCuts) and the Function Discriminant (MethodFDA)). The SA algorithm features two approaches, one starting at minimal temperature (ie, from within a local minimum), slowly increasing, and another one starting at high temperature, slowly decreasing into a minimum. Code developed and written by Kamil Bartlomiej Kraszewski, Maciej Kruk and Krzysztof Danielowski from IFJ and AGH/UJ, Krakow, Poland.

• Plugin capability: custom multivariate classifier can now be plugged into the TMVA framework to benefit from TMVA's analysis and performance comparison tools. The user needs to derive the custom class from TMVA::MethodBase and implement the (few) virtual methods required by the TMVA::IMethod interface. The classifier can then be directly called via ROOT's plugin mechanism. An example for this is given in TMVA/macros/TMVAnalysis.C. Many thanks to Daniel Martschei and Thomas Kuhr (Karlsruhe U.) for suggesting and implementing this feature.

• Preselection cuts now work on arrays. Previously used TEventlists (only event wise pass/fail) were replaced by TreeFormulas (sensitive to array position). Thanks to Arnaud Robert (LPNHE) for his contributions.

• Framework/dataset preparation: Signal and background trees can now be assigned individually to training and test purposes. This is achieved by setting the third parameter of the Factory::AddSignalTree/AddBackgroundTree() methods to "Train" or "Test" (const string). The only restriction is that either none or all signal (background) trees need to be specified with that option. It is possible to mix the two modes, for instance one can assign individual training and test trees for signal, but not for background.

• For increased flexibility, users can also directly input signal and background, training and test events to TMVA, instead of letting TMVA interpret user-given trees. Note that either one of the two approaches must be chosen (no mix). The syntax of the new calls is described in the macros/TMVAnalysis.C test macro. --> The User runs the event loop, copies for each event the input variables into a std:vector, and "adds" them to TMVA, using the dedicated calls: factory->AddSignalTrainingEvent( vars, signalWeight ); (and replacing "Signal" by "Background", and "Training" by "Test"). After the event loop, everything continues as in the standard method.

• Cut optimisation: added physical limits to min/max cuts if smart option is used.

• TMlpANN: fixed crash with ROOT>=5.17 when using large number of test events; also corrected bias in cross validation: before the test events were used, which led to an overestimated performance evaluation in case of a small number of degrees of freedom; separate now training tree in two parts for training and validation with configurable ValidationFraction
  Extended options to TMultilayerPerceptron learning methods.

• BDT: removed hard-coded weight file name; now, paths and names of weight files are written as TObjStrings into ROOT target file, and retrieved for plotting; available weight files (corresponding to target used) can be chosen from pop-up GUI.
  Changes in handling negative weights in BDT algorithm. Events with negative weights now get their weight reduced (*= 1/boostweight) rather than increased (*= boostweight) as the other events do. Otherwise these events tend to receive increasingly stronger boosts, because their effects on the separation gain are as if background events were selected as signal and vice versa (hence the events tend to be "wanted" in signal nodes, but are boosted as if they were misclassified). In addition, the separation indices are protected against negative S or S+B returning 0.5 (no separation at all) in case that occurs.
  In addition there is a new BDT option to ignore events with negative event weights for the training. This option could be used as a cross check of a "worst case" solution for Monte Carlo samples with negative weights. Note that the results of the testing phase still include these events and are hence objective.
  Added randomised trees: similar to the "Random Forests" technique of Leo Breiman and Adele Cutler, it uses the "bagging" algorithm and bases the determination of the best node-split during the training on a random subset of variables only, which is individually chosen for each split.
  Move to TRandom2 for the "bagging" algorithm and throw random weights according to Poisson statistics. (This way the random weights are closer to a resampling with replacement algorithm.)

• GUI: New macro (and GUI button) for Parallel Coordinate plotting.

• Python (PyROOT): added example for reader application: TMVA/python/TMVApplication.py

• Bug fixes:

• Corrected inconsistency in MethodCuts: the signal efficiency written out into the weight file does not correspond to the center of the bin within which the background rejection is maximised (as before) but to the lower left edge of it. This is because the cut optimisation algorithm determines the best background rejection for all signal efficiencies belonging into a bin. Since the best background rejection is in general obtained for the lowest possible signal efficiency, the reference signal efficiency is the lowest value in the bin.

• Fixes in input-variable and MVA plotting: under/over-flow numbers given on plots were not properly normalised; the maximum histogram ranges have been increased to avoid cut-offs. Thanks to Andreas Wenger, Zuerich, for pointing these out.

• Cleaned up memory leaks.

• Added printouts to "Cuts" classifier quoting the explicit cut application for given signal efficiency. In case of transformations of the input variables, the full expressions are given. Added warning to Fisher in case of variable normalisation.

• Macro fixes for ROOT-integrated version.

• Version integrated into ROOT 5.18.

• Improved "mvaeffs.C" macro for significance calculation with arbitrary numbers of signal and background events. This concerns button (5a) of the main GUI (TMVAGui.C).

• Improved "BDT.C" macro for decision tree drawing. This concerns button (8) of the main GUI (TMVAGui.C).

• A few minor GUI upgrades.

• Multiple internal changes (transparent for the users) to prepare ROOT 5.18 production release.

• Bug fixes:

• Fix in MLP network plotter.
• Fixes of smaller memory leaks.
• This version will be integrated in the forthcoming ROOT release 5.18.

• Bug fixes:

• Reimplement possibility to cut on variables that are not used in the MVAs during training and test tree preparation (feature destroyed inadvertedly with latest "bug fix", thanks to Lianyou Shan for spotting this!).

• New normalisation option "NormTree" for the binary search tree used in PDERS. The tree sorting is modified to achieve trees with equal branch lengths, which, in average, speeds up searches. Implementation by Maciej Kruk, Cracow.
• Bug fixes:

• Fix of problem with TChain treatment (occurred in ROOT >=5.15) in DataSet. Thanks to Alfio Rizzo for spotting this!.

• Bug fixes:

• Memory leaks in the Reader class are removed: the Reader is now properly destructed (deletion of all handled classifiers). Thereby, pointer problems in the destructors of Fisher and SVM have been found and fixed.
• In later ROOT versions (>5.14) the macro TMVApplication.C produced a segmentation fault when run from the ROOT prompt (the compiled version in the examples directory worked fine). This problem is now solved.
• The color selection has been adapted to the new color palette that was introduced in ROOT 5.16. The macros should now look alike with all ROOT versions (above 4.02/00).

• The somewhat hastily built release 3.8.8 suffers from an incompatibility with ROOT >= 5.15, leading to segmentation faults in some applications. We have identified the problems and are preparing a new bug-fix release (3.8.9), which should come out on Oct 3.

• Cuts can now be applied independently for signal and background in the PrepareTrainingAndTestTree phase.
• Previously, the input variables used by the Fisher classifier were always normalised to [-1,1] by default. This has been removed, so that it is now in the hand of the user to decide whether or not normalisation is applied. Choose "Normalise" ("!Normalise") for normalisation (no normalisation), default is "!Normalise".
• Bug fixes:

• Major bug fix: the application of cuts in the PrepareTrainingAndTestTree call in conjunction with the use of several trees (ie, several consecutive calls of factory->AddSignalTree(...) or factory->AddBackgroundTree(...)), lead to a wrong application of the cut to all trees but the first one in the signal and background chains. More details can be provided if requested - please contact the authors. We wish to thank Manfred Groh for spotting and analysing the problem!
• Some compilers complained about a missing #include "TMVA/Configurable.h" in the Reader class. This has been fixed.

• Significant speed improvements for PDERS.
  For the options to benefit from this, see the example "PDERSkNN" in macros/TMVAnalysis.C or examples/TMVAnalysis.cxx. Thanks to Kamil Kraszewski and friends from Cracow for implementing this.
• Re-established backward compatibility of TMVA code down to ROOT version 4.02/00.
• Shortened BDT weight-file and standalone C++ reader class by 20% and 50%, respectively.
• Bug fixes:

• Fixed problem in RuleFit's standalone class when using integer input variables.
• Fixed compilation problem when using decorrelation preprocessing of input variables in C++ standalone reader classes.
• Fixed bug in number-of-plots calculation in correlation script.
• Fixed bug in printing of number of events in case of several trees (no impact on results).
• Fixed inconsistency between cut optimisation and cut reading: the aligned definition of min and max cuts is: a variable passes a cut if: min < var <= max. (This inconsistency may have affected your results if you used cut optimisation together with integer variables. Please check with the new version.)
• Fixed macro path in TMVAGui.C to fix problem when running the GUI in the ROOT/TMVA distribution.
  Also: TMVA Style moved from TMVAlogon into tmvaglob to fix style problem when running in the ROOT/TMVA distribution.

• Weight expressions can now be set individually for signal and background via the calls factory->SetSignalWeightExpression( "<signal-expression>" ) and factory->SetBackgroundWeightExpression( "<background-expression>" ). The former call is still supported.
• Overtraining test: a new GUI button (corresponding to an extension of the macro "mvas.C") is available to plot a comparison of the classifier response distributions for the training and independent test data sets. The results of a Kolmogorov-Smirnov compatibility test are printed on stdout and plots.
• The cuts corresponding to a given signal efficiency can be retrieved via the reader.
  An example for this is implemented in "macros/TMVApplication.C". Briefly, retrieve the cuts classifier object as follows: TMVA::MethodCuts* mcuts = (TMVA::MethodCuts*)reader->FindMVA( "CutsGA method" );, define cut vectors (a vector of pairs can also be retrieved via overloaded GetCuts function): std::vector cutsMin; std::vector cutsMax; and fill them via: mcuts->GetCuts( wantedSignalEfficiency, cutsMin, cutsMax );
• Bug fixes:

• Fixed typos in weight file names in MLP and BDT macros
• A comment on code size proliferation: for an intermediate period we are developing a new, more flexible TMVA design in the TMVA/reader repository (for simplicity, to avoid cvs branching). This entails code duplication, which will be removed once the new design is deployed. The duplication has been removed for the tgz-download version.

• Bug fixes:

• Fixed "MinMax" and "RMS" options of PDERS (thanks to Junpei Maeda for spotting this)
• Fixed compilation problem in MetricEuler class on some platforms
• Known problems:

• A filename typo in the ROOT script "macros/network.C" leads to an abort when plotting the MLP architecture. It can be fixed by replacing line 169 by:
  TString fname = "weights/TMVAnalysis_MLP.weights.txt";
  
  The same typo exists in "macros/BDT.C" (plotting a decision tree) where line 8 should be changed into:
  void BDT(Int_t itree=1, TString fname= "weights/TMVAnalysis_BDT.weights.txt", Bool_t useTMVAStyle = kTRUE);
  
  Thanks to Alfio Rizzo and Roger Haelg for reporting these bugs. They have been fixed in CVS and will be distributed with the forthcoming release.

• Added new option for event weight renormalisation to "PrepareTrainingAndTestTree" configuration.
  Possible settings are "None", "NumEvents", "EqualNumEvents", where the latter two options renormalise the event weights so that their sums equal the number of signal and background events, or the number of signal events, respectively. The current version of the TMVA Users Guide has been updated.
• Bug fixes:

• Fixed bug in weights read back of CFMlpANN -> now fully operational again.
• Fixed convergence test macro for MLP neural network.

• Version 4 of the TMVA Users Guide is available: Download PDF file (1.2 MB)
  The guide describes the TMVA classifiers and features, and provides code examples for the interaction with the TMVA Factory and Reader. It corresponds to v3.8.3. To cite it, please refer to arXiv::physics/0703039.
• TMVA-v3.8.3 contains a large number of new features. Please, take the time to digest this announcement. All new classifiers and features are also described (and in much more detail) in V4 of the Users Guide.
• New Classifiers:

• Genuine k-Nearest Neighbour (k-NN) classifier, implemented by Rustem Ospanov (U. of Texas, USA). The classifier is similar to PDERS, but intrinsically adaptive and faster.
• Function Discriminant Analysis (FDA), implemented by Peter Speckmayer and Andreas Hoecker. This simple classifier fits any user-defined TFormula (via option configuration string) to the training data by requiring a formula response of as close as possible to 1 (0) for signal (background) events.
  The parameter fitting is done via the abstract class FitterBase (see below), where the concrete fitter is selected via the configuration option string.
• Upgrades for existing classifiers:

  • Likelihood now allows now to select the interpolation procedure for the reference distributions individually for each variable.
    That is, e.g., spline for var1 Kernel density estimator for var2 etc. In this context, the naming of the options has been changed. While before one would write "Spline=2" or "useKDE", it is now consistently done by: "PDFInterpol=Spline2" or "PDFInterpol=KDE". The previous syntax chooses it for "ALL variables", but one can also specify individually by writing: "PDFInterpol[0]=Spline2:PDFInterpol[1]=KDE" etc.
  • RuleFit has been improved: the forest now consists of boosted trees; a significant speed enhancement has been achieved, and the configuration options have been reorganised.
  • Change of default options in BDT: if no minimum number of events in leaf node is specified, max(20, N_train/(N_cuts^2)/10)) is used.

• Creation of standalone C++ reader classes: each classifier (except for PDERS, k-NN and Cut optimisation) writes together with its weight file a TMVA-independent and ROOT-independent C++ reader class into the same directory as the weight files, and following the same naming convention.
  The classes allow users who need ROOT independence to interpret the trained classifiers. They provide identical results as the TMVA-integrated Reader. The example script TMVA/macros/ClassApplication.C shows how to use the new classes.
• Classifier-specific help messages: when invoking the "H" option in the configuration option string, each classifier prints a help message with a short summary of the working principles, and, more importantly, hints on which options to tune to improve selection and/or speed performance.
  The help messages of all booked classifiers can also be printed by invoking the factory via the call: factory->PrintHelpMessage().
• The various TMVA fitters, used for minimisation problems by the classifiers, have been upgraded and unified under the common base class FitterBase.
  Any TMVA module that uses minimisation inherits from IFitterTarget, which interfaces the minimisation function. Presently, four concrete fit algorithms are implemented: Monte Carlo sampling (MC), Genetic Algorithm (GA), MINUIT and Simulated Annealing (SA not yet finalised). Also allowed are combinations of fitters, such as using MC or GA together with MINUIT for better convergence under starting values provided by the former algorithms.
• Change of the somewhat indigestive mu-transform into the Rarity variable. The rarity is a variable transformation that makes the background uniform distributed in [0,1], while the signal is peaking towards 1.
  It can be also useful to spot simulation (or control sample) problems: if the background in data is not flat this would hint to problems in the background description. A useful variable, try it out!
• Other changes:

• Internal renormalisation of event weights: to avoid artefacts if event weights have very small or large numbers, we renormalise the sum of weights of all signal and background training events to be equal the number of events. This does not affect the relative event weights.
• New directory structure for histograms in the target file produced by the training. The classifier directories have subdirectories according to the classifier names. This allows to better organise the histograms produced by different instances of the same classifier (e.g., during optimisation scans of that classifier). The GUI correctly uses the new structure. Some changes also occur in the weight files, though backward compatibility is guaranteed through the TMVA versioning.
• Change in TMVA::Config class: the structs with global settings can no longer be addressed via (e.g.): , TMVA::gConfig().ionames.weightFileDir = "myWeightDirectory"; but rather via: (TMVA::gConfig().GetIONames()).fWeightFileDir = "myWeightDirectory"; This change is not backward compatible, but only concerns script commands, which should be easily updatable.
• Bug fixes:

• Full fix of treatment of integer variables in Reader application (thanks to Andrea Bulgarelli for spotting this bug in the first place).
• Fix of potential branch name/title mismatch in DataSet (bug spotted by Zhiyi Liu).
• Known problems:

• The performance of the CFMlpANN neural network looks questionable. We did not yet investigate whether there is a real problem, but for the users who apply this ANN please either move the the recommended MLP ANN, or contact the authors.
• The combined use of GA or MC with a Minuit converger in FDA has a problem in the configuration and is thus inoperational at present.

• A serious bug has been discovered in the TMVA::Reader when dealing with 'I'-declared (int) variables. In that case, the Reader response is inconsistent with the test results derived in the training phase, and wrong (the test results are correct). We have uploaded TMVA-v3.7.3 with a fix of this bug - with however the temporary restriction that decorrelation preprocessing in presence of 'I'-declared variables does not work. We are working on a full fix of the problem.
  

  There are two possibilities to circumvent the bug:

  1. Upgrade your TMVA version to v3.7.3, declare as usual your integer variables as 'I', but avoid using decorrelation or PCA transformation.
  2. Do not upgrade TMVA, use integer variables, but DO NOT declare these variables as 'I', but as 'F', when adding them to the Factory. In addition, when using the Reader, declare all variables that are added to the Reader as float, and just copy the integer variable into this float placeholder. All results will be correct (only Likelihood will treat these as continuous variables, but the performance should not be significantly affected by this).

April 10-18, 2007 (TMVA-v3.7.0 - v3.7.2)

• Version 3 of the TMVA Users Guide is available: Download PDF file (1.2 MB)
  
  The guide describes the TMVA classifiers and features, and provides code examples for the interaction with the TMVA Factory and Reader. It corresponds to v3.7.2. To cite it, please refer to arXiv::physics/0703039
• Classifier updates and improvements:
• New Support Vector Machine (SVM) classifier, implemented by Andrzej Zemla and Marcin Wolter (IFJ PAN Krakow, Poland). Currently it is still a beta version, which is functional but has not yet been exhaustively tested. Feedback is welcome. Fours different kernel functions are available: linear, polynomial, Gaussian and sigmoidal. SVM performs well under our standard toy example, but not many more applications have been tried so far.
• Reduced memory consumption of PDERS.
• Speed up of Fisher and H-Matrix in cases where many (>30) variables are used.
• New features and code updates:
• Where useful, options can be set per variable and event class. An example for this is the creation of reference distributions (PDFs) for the likelihood classifier: each variable, individually for signal and background, can have custom interpolation and smoothing settings.
• Upon request, the weight files are now exclusively written in text (ascii) format. If wished, an option also allows the writing in ROOT format, but this is supposed to be the exception from the rule.
• Input variables that are part of an array or vector are now accepted if specified with the array index. Example: factory->AddVariable("pT[0]",'F').
• Colorful logging output can be switched off using Factory option. In batch mode, colors are switched off by default to avoid flooding the logfiles with ESC sequences.
• Bug fixes:
• New for v3.7.2: fix problem with use of discrete variables in Likelihood classifier.
• Fixed problem in weight file reading for rectangular cut optimisation. Note that the format has slightly changed, which will affect the reading of previous weight files (for Cuts only).
• Discrete variables are now correctly plotted (in variables.C).
• We gratefully acknowledge the many valuable inputs and comments from watchful users!

• Significant speed enhancement for RuleFit. Also improved robustness of RuleFit response
• Individualised likelihood options: smoothing parameters and number of bins in reference histograms of binned non-parametric splines can now be set for each variable and class (signal, background)

• The complete TMVA Users Guide is now available: Download PDF file (1.2 MB)
  
  The guide describes the TMVA classifiers and features, and provides code examples for the interaction with the TMVA Factory and Reader. It corresponds to v3.6.0.
• Classifier updates and improvements for v3.6.0:
• New adaptive unbinned kernel density estimator (KDE) for the projective likelihood classifier (thanks to Asen Christov).
• Ranking of input variables for projective likelihood classifier.
• New analysis macros for RuleFit. Other RuleFit developments to improve its robustness.
• New features and code updates for v3.6.0:
• Random and block-wise shuffling of training and test events (optional).
• New interface for the registration of trees for training and testing allowing unequal numbers of training and testing events for signal and background. The interface is backward compatible.
• Branches of the user TTree not used to compute the input variables are automatically disabled to speed up the training and testing tree preparation.
• On user request (option CreateMVAPdfs) TMVA creates probability density functions for the classifier signal and response distributions using the training data. These can then be retrieved via the reader method: reader->GetProba( "" ) (thanks to Sophie Henrot-Versillé).
• All classifiers now write ROOT and text weight files. The ROOT files contain (at least) PDFs for the classifiers. Given the need by some collaborations to store weight files in conditions databases, we are considering to move to text-only formats for the next release. Please contact us if this may create incompatibilities.
• New Configurable base class simplifying the configuration of Factory and classifiers.

• Reorganisation (and decluttering) of treatment of variable transformations prior to applying MVA data mining (these transformations were previously called "preprocessing"). Although this upgrade corresponds to significant code evolution, most of it does not affect the user.
• Upgrade of result plots and macros.
• Further performance improvements in RuleFit.
• Few other insignificant and cosmetics changes.

• Many internal changes regarding convenience and robustness of TMVA. In particular, the automatic decorrelation of the input variables, which led to problems in the past, has been turned off. Decorrelation (in general: data preprocessing) is now performed on demand (by option of the Factory, or whenever a methods requires it).
• Much improved BDT pruning. Also automatic detection of the pruning strength is available, which is however not performed by default: use the option string to change defaults.
• TMVA can also be run from python using PyROOT. An example code can be found in: TMVA/python/TMVAnalysis.py. Execute it from the command line by calling (from the TMVA/python/ directory): python TMVAnalysis.py --method "Fisher Likelihood", or any other methods. Type python TMVAnalysis.py --help, for the usage.

• Some internal code changes, such as the introduction of a message logger to replace standard output (you will notice the nicely formatted output, thanks to Attila Krasznahorkay).
• Bug fixes in the macros for the result plotting. Previously, the TMVAGUI used to crash after a few plotting iterations.
• Improvements in the BDT pruning.

• A few outstanding problems have been fixed. Among these: input data preprocessing (such as decorrelation) is now performed on-the-fly (before, individual trees were created in the DataSet). This allows the MVAs to obtain the appropriate preprocessing separately for signal and background (as necessary, e.g., in MethodLikelihood).

• MVA methods:
  • The multi-dimensional PDE range-search method has been extended by a larger choice of kernel functions (Gaussian, teepee (triangular), Sinc and Lanczos). Moreover, instead of rectangular hypervolumes, one can also use hyperellipsoides to define the reference volume. Many tests indicate that large volume sizes perform better. Please send comments/questions to Yair Mahalalel (CERN)
  • We provide a first release of a new implementation of RuleFit (see paper by Friedman-Popescu). We note that its state is still preliminary. Some more testing is needed to understand and optimise all the features of this method, and to find a good default configuration. Please send comments/questions to Fredrik Tegenfeldt (Iowa State U. till Aug 2007).
  • Boosted Decision Trees (BDT) can now be pruned. The corresponding option is PruneStrength=10, setup to a default value of 10. Pruning removes decision-tree nodes that are considered noise.

• New analysis macros:
  • The macro ../macros/annconvergencetest.C displays the convergence of the MLP network for training and test events. It is called pressing the new button "Network Convergence Test" of the GUI: root -l TMVAGui.C\(\"TMVA.root\"\).

• New methods under active development are: Support Vector Machine and a combined (ruled) use of various MVAs in a committe of methods.

• The next full update of the ROOT version of TMVA will go into the ROOT production release v5.13 (December 2006).

• TMVA has been undergone a major redesign and upgrade during the Summer 2006. Most of the changes are hidden to the TMVA user. The changes/new features that do affect the users are listed below:

• Job configuration as far as it concerns the user:
  • The declaration of MVA input variables allows to specify the variable type (however, the only method that currently uses type information is Likelihood).
  • Global (tree) and event-by-event weights are now included. All TMVA methods but Cuts, Fisher and HMatrix support event weights (for these methods they are ignored).
  • The method booking interface has slightly changed: several bookings of the same method (different instances) are now possible. This allows to train and test a method with different configurations.
  • The option string provided by the user during the booking phase has been enhanced: options are now given independent of the order, and the interface is made clearer.
  • The weight files are now entirely self-contained, ie, they contain the full configuration information and training results (weights). When using a trained method in the Reader, it is sufficient to give "some" method name and the corresponding weight file.
  • Formula expressions (functions of variables) can be used as input variables.
  • TChains and TTrees are supported as data inputs for training.
  Please see the usual example macros (macros/TMVAnalysis.C and macros/TMVApplication.C) to lookup concrete implementations of the new interfaces.

• Training and data preparation features:
  • Linear decorrelation of input variables is centrally provided by the new DataSet class, and is available to all TMVA methods (particularly interesting for Cut optimization and Likelihood). Add ":D" to the option string.
  • Most of the methods now return (and print to stdout) ranked input variables, according to the classification criteria used by the method.

• MVA methods:
  • New artificial neural network: MLP (implemented by Matt Jachowski, Stanford U). This fast and powerful multilayer perceptron is the recommended TMVA neural network. We have tested that MLP performs at least as well as TMlpANN, and significantly better than CFMlpANN.
  • Correct treatment of discrete variables in Likelihood.
  • Boosted decision trees (BDT) come now with pruning.
  • The Genetics Algorithm for Cuts optimization has been significantly improved.

• New analysis macros:
  • Simple GUI for results plotting (macros/TMVAGui.C) to be called after training and testing in "macros" dir via the prompt command:
    root -l TMVAGui.C\(\"TMVA.root\"\)
  • Scatter and profile plots to better visualise correlations between input variables
  • Quality verification for Likelihood reference distributions.
  • Visualisation of trained MLP network architecture.

• Known problems/bugs (see also the developers 'todo' list):
  • BDT cannot run on a sample of solely discrete variables.
  • Decorrelation does not distinguish signal and background (signal is used).
  • Event weights are not yet consistently treated in all TMVA plots. We think (hope) that this does not have an impact on the MLP, BDT and Likelihood event-wise weighting, and hence performance.

• Forthcoming MVA method developments: we are currently actively working on the improvement of PDERS and BDT; new methods under active development are: Support Vector Machine and RuleFit. We are also developing combined (ruled) use of various MVAs in a committee.

• The next full update of the ROOT version of TMVA will go into the ROOT production release v5.13 (December 2006).

• TMVA has been integrated in ROOT 5.11/06 (development version). The full TMVA functionality is available therein. For demonstration purpose, the macro TMVAnalysis.C is provided in the tutorial section. More information at the following links:
• ROOT release notes
• ROOT::TMVA class index
• ROOT download page
• For obvious reasons the ROOT-integrated version is not uptodate with the cvs HEAD that can be downloaded by users from sourceforge. New features available here include
  • Simulated Annealing fitter for cut optimisation (needs still some tuning)
  • TFormula-parsed operational strings can be used as MVA input variables