A common problem in particle physics is to employ a NN for particle identification (e.g. “proton”, “positron” or “pion”, etc. ). A very similar problem is one in which we are trying to train a NN discriminator to differentiate between signal and background classes of events by using information about the underlying physics contained in kinematic variables, e.g. from particles colliding at an accelerator. This is a binary (and supervised) classification problem, i.e. the classification has only two possible discrete outcomes. Such problems in particle physics tend to be multi-dimensional and non-linear, with the NN requiring a large number of inputs (i.e. the kinematic variables). In this project, I will be using simulated data from the ATLAS experiment at CERN. I will be employing NN techniques for classification in a search for exotic particles, and evaluate the performance of these architectures.
There are many exotic models that predict new exotic (also known as “Beyond the Stan- dard Model”) particles. Searches for new particles tend to focus on the large-mass region of the phasespace, as the assumption is that lighter particles would have been discovered in searches at previous experiments at lower collision energies.
In this project, I will be considering a search for an (additional) heavy Higgs boson decaying into pairs of vector bosons V (where V = W or Z), see the figure below. Hadronic decays of vector bosons have large branching fractions but also large backgrounds, whereas leptonic decays come with lower branching fractions and a cleaner experimental signature. A compromise between large backgrounds and very small signal yields is to require one of the vector bosons to be a Z decaying leptonically (Z → l+l−, with l = e or μ), with the second vector boson decaying hadronically (V → qq′)1. One would expect the hadronisation of the two quarks to give two distinct reconstructed jets of particles inside the detector. However, for V bosons originating from very heavy particles (in the TeV region) encountered in typical exotic-search signatures at the Large Hadron Collider (LHC), the quarks from the hadronic V decay tend to partially overlap in the η − φ space, giving rise to an experimental signature that requires special treatment. Searches that include this boosted, partially-overlapping quark hadronisation use the so- called “fat” (large-R) jet reconstruction (J). The aim is to differentiate this experimental signature from the one of single-quark or gluon hadronisation, typically captured with the baseline jet reconstruction algorithm (j).
Figure 1: Representative Feynman diagram for the production of a heavy resonance X via gluon fusion (incoming gg pair), with its subsequent decay into a pair of vector bosons (specifically: ZZ in this diagram). In the example of the X decay into a pair of Z bosons, this analysis described in the text would be sensitive in the final state in which one of the Z bosons decays leptonically, with the second one decaying hadronically.
In any physics analysis, one of the most important ingredients is the characterisation of the experimental signature of the final state with a list of decay products. For the search of heavy Higgs bosons, the chosen final state here is two opposite-sign charged leptons (l+l−, with l = e or μ), and one fat-jet (J).
Additional requirements may be that the invariant mass of the charged leptons is consistent with the Z boson mass (mll ≃ 91 GeV/c2), or that the invariant mass of the dilepton pair and the fat jet are consistent with the Higgs boson mass2. The next task is to identify potential sources of SM background processes that could mimic the signal of interest. The most obvious one is the non-peaking diboson contin- uum, qq → Z(ll)V (qq′). We should also add the inclusive Z production, qq → Z(ll)+ jets. Finally, we should include the tt ̄ production, since the leptonic decays of the W bosons from the t → W b decays can lead to final states with two charged leptons (and occasionally boosted jets).
I will be working with simulated datasets for all these (signal and background) pro- cesses, containing kinematic information for all decay products of interest.
CSV files with the kinematic information for about 680k background and 50k signal simulated events can be found at https://cernbox.cern.ch/s/pCMUZTKFtVvrxUh. The background events are split into three different files corresponding to different physics processes3. The signal sample has been generated for a hypothetical heavy Higgs boson with mH = 1 TeV/c2. Every entry in these tables corresponds to a separate (signal or background) collision event. The files contain a large number of variables per event. Here we will focus on a subset of them, summarised in Table 1.
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import sklearn
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import Dropout
from tensorflow.keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
Diboson = pd.read_csv('Diboson.csv')
ggh1000 = pd.read_csv('ggH1000.csv')
top = pd.read_csv('Top.csv')
zjets = pd.read_csv('Zjets.csv')
Diboson.head()
Unnamed: 0 | FullEventWeight | MET | Topology | Zll_mass | Zll_pt | fatjet_C2 | fatjet_D2 | fatjet_E | fatjet_eta | ... | lep1_eta | lep1_phi | lep1_pt | lep2_E | lep2_charge | lep2_eta | lep2_phi | lep2_pt | reco_zv_mass | truth_zv_mass | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | 0 | 0.387345 | 41920.540 | 0 | 187549.810 | 374316.600 | 0.127114 | 1.786927 | 353161.47 | -0.001591 | ... | 1.677279 | -1.346853 | 369495.34 | 12147.315 | 1 | -0.686657 | -2.412152 | 9755.2705 | 1014320.10 | -1.0 |
1 | 1 | 0.172328 | 225618.530 | 0 | 13106.807 | 51429.535 | 0.269517 | 2.296220 | 357617.66 | -1.066937 | ... | -2.057147 | 2.975854 | 36972.74 | 34980.440 | -1 | -1.521354 | 2.819377 | 14584.7360 | 240295.97 | -1.0 |
2 | 2 | 0.371392 | 42284.125 | 0 | 84519.230 | 153790.270 | 0.051175 | 3.360730 | 345472.30 | 0.704825 | ... | -0.285897 | -0.084159 | 146895.95 | 24618.990 | -1 | 0.741342 | 1.177780 | 19119.7030 | 471285.40 | -1.0 |
3 | 3 | 0.309902 | 42735.938 | 1 | 76138.640 | 102611.836 | 0.117028 | 0.702838 | 476446.80 | 0.095546 | ... | -0.160614 | -2.544057 | 109699.72 | 22303.861 | 1 | -1.648011 | 1.120479 | 8277.4480 | 495503.10 | -1.0 |
4 | 4 | 0.202772 | 70475.780 | 0 | 86138.000 | 141886.280 | 0.053951 | 3.634348 | 799487.75 | -1.828396 | ... | -0.499056 | 0.429884 | 151453.40 | 14053.401 | -1 | -0.171763 | 2.796952 | 13848.2230 | 498823.40 | -1.0 |
5 rows × 25 columns
kinematics = ['lep1_pt','lep2_pt','fatjet_pt','fatjet_eta','fatjet_D2', 'Zll_mass','Zll_pt','MET','reco_zv_mass','isSignal']
Diboson = Diboson[kinematics]
Diboson.head(10)
lep1_pt | lep2_pt | fatjet_pt | fatjet_eta | fatjet_D2 | Zll_mass | Zll_pt | MET | reco_zv_mass | isSignal | |
---|---|---|---|---|---|---|---|---|---|---|
0 | 369495.34 | 9755.2705 | 345940.12 | -0.001591 | 1.786927 | 187549.810 | 374316.600 | 41920.540 | 1014320.10 | 0 |
1 | 36972.74 | 14584.7360 | 218057.10 | -1.066937 | 2.296220 | 13106.807 | 51429.535 | 225618.530 | 240295.97 | 0 |
2 | 146895.95 | 19119.7030 | 274339.25 | 0.704825 | 3.360730 | 84519.230 | 153790.270 | 42284.125 | 471285.40 | 0 |
3 | 109699.72 | 8277.4480 | 444837.44 | 0.095546 | 0.702838 | 76138.640 | 102611.836 | 42735.938 | 495503.10 | 0 |
4 | 151453.40 | 13848.2230 | 250422.77 | -1.828396 | 3.634348 | 86138.000 | 141886.280 | 70475.780 | 498823.40 | 0 |
5 | 346327.66 | 142251.2500 | 350558.56 | -1.187918 | 1.412536 | 324188.380 | 406563.840 | 50060.074 | 921642.75 | 0 |
6 | 159703.48 | 114831.1950 | 263462.22 | 0.503505 | 2.235607 | 323966.250 | 145375.170 | 149063.600 | 707851.94 | 0 |
7 | 178415.02 | 29447.8610 | 551925.30 | 0.128561 | 1.162044 | 74883.880 | 206981.550 | 101938.516 | 666803.80 | 0 |
8 | 27683.80 | 7713.0990 | 243752.19 | 1.673680 | 1.207476 | 105245.130 | 21670.336 | 68859.695 | 339188.10 | 0 |
9 | 115206.40 | 25674.4200 | 404830.20 | 1.018624 | 0.959880 | 34472.230 | 140847.640 | 91502.730 | 563217.30 | 0 |
ggh1000 = ggh1000[kinematics]
ggh1000.head(5)
lep1_pt | lep2_pt | fatjet_pt | fatjet_eta | fatjet_D2 | Zll_mass | Zll_pt | MET | reco_zv_mass | isSignal | |
---|---|---|---|---|---|---|---|---|---|---|
0 | 320104.62 | 312638.44 | 480641.84 | -0.851387 | 0.889354 | 87475.55 | 627990.80 | 44495.770 | 1063465.80 | 1 |
1 | 288589.78 | 129548.62 | 375352.78 | 1.041311 | 1.485706 | 90507.31 | 409780.94 | 34441.156 | 974210.75 | 1 |
2 | 228625.23 | 118590.28 | 555879.20 | 0.417858 | 1.219367 | 89765.85 | 344668.30 | 21625.379 | 996837.40 | 1 |
3 | 302502.84 | 233439.89 | 430646.20 | 0.438590 | 0.592705 | 92313.63 | 535629.06 | 21855.973 | 989655.56 | 1 |
4 | 224785.02 | 76205.98 | 244348.40 | 1.213717 | 0.724910 | 89451.92 | 296257.47 | 33677.380 | 993398.30 | 1 |
top = top[kinematics]
top.head(5)
lep1_pt | lep2_pt | fatjet_pt | fatjet_eta | fatjet_D2 | Zll_mass | Zll_pt | MET | reco_zv_mass | isSignal | |
---|---|---|---|---|---|---|---|---|---|---|
0 | 38513.980 | 9003.888 | 253407.78 | -1.195378 | 1.435712 | 35253.492 | 35612.324 | 51609.760 | 245723.92 | 0 |
1 | 34191.734 | 9425.637 | 256178.67 | 0.387434 | 2.403266 | 20311.205 | 38644.793 | 225038.800 | 351938.88 | 0 |
2 | 254871.310 | 17218.764 | 282584.72 | -1.822714 | 1.336584 | 134715.500 | 244261.840 | 27090.697 | 1046120.90 | 0 |
3 | 84608.050 | 9801.820 | 205555.95 | -1.725750 | 1.780693 | 69086.836 | 90045.625 | 57572.890 | 301995.70 | 0 |
4 | 50720.953 | 10932.975 | 256003.56 | 1.477704 | 3.398984 | 51731.508 | 52780.684 | 123177.900 | 376986.28 | 0 |
zjets = zjets[kinematics]
zjets.head(5)
lep1_pt | lep2_pt | fatjet_pt | fatjet_eta | fatjet_D2 | Zll_mass | Zll_pt | MET | reco_zv_mass | isSignal | |
---|---|---|---|---|---|---|---|---|---|---|
0 | 76128.370 | 11973.011 | 210397.60 | -1.329446 | 1.895164 | 95697.766 | 64252.690 | 177227.420 | 596020.44 | 0 |
1 | 40799.945 | 27624.867 | 226681.30 | -0.429050 | 0.000000 | 90289.914 | 32506.705 | 10660.757 | 260357.19 | 0 |
2 | 57988.660 | 27102.590 | 547212.70 | -0.543789 | 2.463149 | 92994.430 | 55287.625 | 655334.400 | 422698.44 | 0 |
3 | 55424.742 | 51873.030 | 200684.47 | 1.776149 | 2.081222 | 89607.580 | 63405.484 | 41489.973 | 302718.06 | 0 |
4 | 68312.220 | 28290.979 | 324990.97 | -1.524169 | 2.321676 | 90199.480 | 47187.332 | 41731.630 | 461896.25 | 0 |
# Plotting the distributions
for i in range(9):
##Remove the largest 200 data points
D_data=Diboson[kinematics[i]].sort_values()[:-200]
G_data=ggh1000[kinematics[i]].sort_values()[:-200]
T_data=top[kinematics[i]].sort_values()[:-200]
Z_data=zjets[kinematics[i]].sort_values()[:-200]
#Plotting the normalised histograms with 100 bins
plt.hist(D_data,density=True,label='Diboson Background',bins=100)
plt.hist(G_data,density=True,label='Higgs Signal',bins=100)
plt.hist(T_data,density=True,label='Top Quark Background',bins=100)
plt.hist(Z_data,density=True,label='Z-jets Background',bins=100)
plt.title("Normalised Histogram of {}".format(kinematics[i]))
plt.legend()
plt.show()
C:\Users\boazm\AppData\Local\Temp\ipykernel_24460\1938706090.py:5: FutureWarning: The behavior of `series[i:j]` with an integer-dtype index is deprecated. In a future version, this will be treated as *label-based* indexing, consistent with e.g. `series[i]` lookups. To retain the old behavior, use `series.iloc[i:j]`. To get the future behavior, use `series.loc[i:j]`. D_data=Diboson[kinematics[i]].sort_values()[:-200] C:\Users\boazm\AppData\Local\Temp\ipykernel_24460\1938706090.py:6: FutureWarning: The behavior of `series[i:j]` with an integer-dtype index is deprecated. In a future version, this will be treated as *label-based* indexing, consistent with e.g. `series[i]` lookups. To retain the old behavior, use `series.iloc[i:j]`. To get the future behavior, use `series.loc[i:j]`. G_data=ggh1000[kinematics[i]].sort_values()[:-200] C:\Users\boazm\AppData\Local\Temp\ipykernel_24460\1938706090.py:7: FutureWarning: The behavior of `series[i:j]` with an integer-dtype index is deprecated. In a future version, this will be treated as *label-based* indexing, consistent with e.g. `series[i]` lookups. To retain the old behavior, use `series.iloc[i:j]`. To get the future behavior, use `series.loc[i:j]`. T_data=top[kinematics[i]].sort_values()[:-200] C:\Users\boazm\AppData\Local\Temp\ipykernel_24460\1938706090.py:8: FutureWarning: The behavior of `series[i:j]` with an integer-dtype index is deprecated. In a future version, this will be treated as *label-based* indexing, consistent with e.g. `series[i]` lookups. To retain the old behavior, use `series.iloc[i:j]`. To get the future behavior, use `series.loc[i:j]`. Z_data=zjets[kinematics[i]].sort_values()[:-200]
The graphs most of the parameters are useful for seprarting the signal events from the background events. Thw worst parameters are the fatjet_eta and the MET. The best parameters are the rec0_zv_mass and the transverse lepton momenta.
mega_background = pd.concat([Diboson,top,zjets],ignore_index=True) # Concatenate the background dataframes
mega_background = sklearn.utils.shuffle(mega_background,random_state=144) # shuffing
mega_background = mega_background.reset_index(drop=True) # reseting the index
mega_background
lep1_pt | lep2_pt | fatjet_pt | fatjet_eta | fatjet_D2 | Zll_mass | Zll_pt | MET | reco_zv_mass | isSignal | |
---|---|---|---|---|---|---|---|---|---|---|
0 | 129987.200 | 129684.125 | 212218.42 | 0.059829 | 4.130715 | 93298.880 | 252736.470 | 30111.9020 | 525693.80 | 0 |
1 | 275281.200 | 142238.560 | 304447.80 | -0.054227 | 2.037450 | 308534.900 | 293354.750 | 8518.9890 | 737451.56 | 0 |
2 | 73241.086 | 27316.834 | 539494.70 | 0.534243 | 2.600273 | 89771.680 | 48323.970 | 66358.9800 | 384654.80 | 0 |
3 | 80157.590 | 32266.120 | 351070.94 | 0.091386 | 1.362572 | 102161.780 | 47891.906 | 34213.7930 | 446084.40 | 0 |
4 | 317650.970 | 22914.078 | 303283.78 | 0.414271 | 2.004709 | 82843.664 | 340156.280 | 2285.5596 | 678935.60 | 0 |
... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
6826954 | 234206.360 | 38218.960 | 352723.56 | 0.554933 | 3.040847 | 92949.360 | 269912.220 | 25515.9260 | 637996.06 | 0 |
6826955 | 232562.450 | 8332.054 | 264437.00 | 1.351408 | 2.432621 | 91500.770 | 226494.420 | 11184.7670 | 507551.47 | 0 |
6826956 | 398864.250 | 78154.240 | 618088.00 | 1.565232 | 2.509288 | 231143.380 | 444340.530 | 97790.8360 | 1189955.40 | 0 |
6826957 | 150969.660 | 110124.250 | 221963.19 | -1.370130 | 1.805588 | 96038.480 | 243111.900 | 20369.0760 | 619579.56 | 0 |
6826958 | 180550.770 | 31216.906 | 227068.64 | 1.969358 | 2.501968 | 130939.800 | 187642.670 | 21793.4840 | 499331.12 | 0 |
6826959 rows × 10 columns
# Add the signal dataframe
xtrain = pd.concat([ggh1000,mega_background[:len(ggh1000)]],ignore_index=True)
xtrain = sklearn.utils.shuffle(xtrain,random_state=144)
xtrain = xtrain.reset_index(drop=True)
xtrain
lep1_pt | lep2_pt | fatjet_pt | fatjet_eta | fatjet_D2 | Zll_mass | Zll_pt | MET | reco_zv_mass | isSignal | |
---|---|---|---|---|---|---|---|---|---|---|
0 | 311621.44 | 166917.380 | 351686.66 | -0.893446 | 1.296051 | 97003.766 | 476325.030 | 100914.5200 | 869298.60 | 1 |
1 | 371676.44 | 56707.793 | 510651.66 | 0.360803 | 1.138071 | 89759.650 | 422098.100 | 32114.2660 | 944189.10 | 1 |
2 | 183726.81 | 96609.980 | 306309.72 | -1.528060 | 1.639424 | 89656.970 | 279330.200 | 9913.6620 | 951625.30 | 1 |
3 | 395374.40 | 77077.800 | 475170.12 | 0.482761 | 0.829896 | 93623.890 | 472317.160 | 7209.2800 | 1012361.75 | 1 |
4 | 113136.06 | 105196.600 | 215116.28 | -0.665583 | 2.144964 | 86765.620 | 209741.920 | 22493.1210 | 537172.56 | 0 |
... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
100021 | 285666.53 | 179056.270 | 494486.70 | -0.009768 | 1.379898 | 128783.480 | 457582.780 | 11007.6875 | 1031443.00 | 1 |
100022 | 187699.80 | 9407.155 | 264273.50 | 0.996903 | 2.373407 | 85700.070 | 178307.160 | 15041.1980 | 511582.38 | 0 |
100023 | 188336.95 | 57431.520 | 443256.70 | 0.008493 | 2.157738 | 85286.730 | 240943.580 | 49264.6560 | 1051562.50 | 0 |
100024 | 238174.58 | 195302.470 | 492718.78 | 0.536637 | 0.347302 | 85085.290 | 431991.380 | 67326.5800 | 999951.30 | 1 |
100025 | 93622.51 | 54618.363 | 246335.67 | 0.210060 | 1.122656 | 92714.080 | 117757.445 | 14416.5980 | 363755.56 | 0 |
100026 rows × 10 columns
#list of input features (minus reco_zv_mass)
input_features = ['lep1_pt','lep2_pt','fatjet_pt','fatjet_eta','fatjet_D2','Zll_mass','Zll_pt','MET']
#all features (minus reco_zv_mass)
features = ['lep1_pt','lep2_pt','fatjet_pt','fatjet_eta','fatjet_D2','Zll_mass','Zll_pt','MET','isSignal']
#xtrain without the reco_zv_mass
xtrain_minus = xtrain[features]
# Create plots for each variable
for i in range(len(input_features)):
for j in range(i+1,len(input_features)): # Avoid double counting
col = np.where(xtrain_minus['isSignal']==1,'r','b')
plt.scatter(xtrain_minus[input_features[i]],xtrain_minus[input_features[j]],color=col,label='Signal = red, background = blue')
plt.title("Graph of {} vs {}".format(input_features[i],input_features[j]))
plt.xlabel(input_features[i])
plt.ylabel(input_features[j])
plt.legend()
plt.show() # signal is red and noise is blue
from sklearn import model_selection,preprocessing
sc = preprocessing.StandardScaler()
input_data = sc.fit_transform(xtrain_minus[input_features])
target = xtrain_minus['isSignal']
# set random seed
Answer_to_all_questions = 42
# train - test split of dataset
train_data,test_data, train_target,test_target = model_selection.train_test_split(\
input_data,target,test_size =0.3,random_state = Answer_to_all_questions)
print(train_data.shape,train_target.shape,test_data.shape,test_target.shape)
(70018, 8) (70018,) (30008, 8) (30008,)
def my_model(num_inputs,num_outputs,num_nodes,extra_depth):
# create model
model = Sequential()
model.add(Dense(num_nodes,input_dim = num_inputs, kernel_initializer = 'normal',\
activation = 'relu' ))
model.add(Dropout(0.2))
for i in range(extra_depth):
model.add(Dense(num_nodes,kernel_initializer = 'normal',activation='relu'))
model.add(Dense(num_outputs,activation = 'sigmoid'))
# Compile model
model.compile(loss ='binary_crossentropy', optimizer = 'adam', metrics =['accuracy'])
return model
batchSize=500
N_epochs=50
num_inputs = 8
num_nodes = 20
extra_depth=1
num_outputs=1
model = my_model(num_inputs,num_outputs,num_nodes,extra_depth)
history = model.fit(train_data, train_target, batch_size = batchSize, epochs = N_epochs, \
verbose = 1, validation_data =(test_data, test_target))
Epoch 1/50 141/141 [==============================] - 2s 7ms/step - loss: 0.4759 - accuracy: 0.8352 - val_loss: 0.3385 - val_accuracy: 0.8714 Epoch 2/50 141/141 [==============================] - 1s 5ms/step - loss: 0.3296 - accuracy: 0.8728 - val_loss: 0.3050 - val_accuracy: 0.8783 Epoch 3/50 141/141 [==============================] - 1s 5ms/step - loss: 0.3108 - accuracy: 0.8793 - val_loss: 0.2892 - val_accuracy: 0.8851 Epoch 4/50 141/141 [==============================] - 1s 5ms/step - loss: 0.3035 - accuracy: 0.8805 - val_loss: 0.2827 - val_accuracy: 0.8872 Epoch 5/50 141/141 [==============================] - 1s 4ms/step - loss: 0.2989 - accuracy: 0.8820 - val_loss: 0.2789 - val_accuracy: 0.8883 Epoch 6/50 141/141 [==============================] - 1s 4ms/step - loss: 0.2974 - accuracy: 0.8820 - val_loss: 0.2766 - val_accuracy: 0.8896 Epoch 7/50 141/141 [==============================] - 1s 5ms/step - loss: 0.2951 - accuracy: 0.8838 - val_loss: 0.2746 - val_accuracy: 0.8902 Epoch 8/50 141/141 [==============================] - 1s 5ms/step - loss: 0.2917 - accuracy: 0.8844 - val_loss: 0.2722 - val_accuracy: 0.8911 Epoch 9/50 141/141 [==============================] - 1s 5ms/step - loss: 0.2912 - accuracy: 0.8852 - val_loss: 0.2714 - val_accuracy: 0.8909 Epoch 10/50 141/141 [==============================] - 1s 4ms/step - loss: 0.2899 - accuracy: 0.8850 - val_loss: 0.2709 - val_accuracy: 0.8911 Epoch 11/50 141/141 [==============================] - 1s 5ms/step - loss: 0.2892 - accuracy: 0.8847 - val_loss: 0.2685 - val_accuracy: 0.8930 Epoch 12/50 141/141 [==============================] - 1s 5ms/step - loss: 0.2878 - accuracy: 0.8854 - val_loss: 0.2691 - val_accuracy: 0.8928 Epoch 13/50 141/141 [==============================] - 1s 4ms/step - loss: 0.2869 - accuracy: 0.8858 - val_loss: 0.2683 - val_accuracy: 0.8932 Epoch 14/50 141/141 [==============================] - 1s 4ms/step - loss: 0.2861 - accuracy: 0.8863 - val_loss: 0.2676 - val_accuracy: 0.8925 Epoch 15/50 141/141 [==============================] - 1s 4ms/step - loss: 0.2852 - accuracy: 0.8860 - val_loss: 0.2665 - val_accuracy: 0.8928 Epoch 16/50 141/141 [==============================] - 1s 4ms/step - loss: 0.2843 - accuracy: 0.8870 - val_loss: 0.2658 - val_accuracy: 0.8936 Epoch 17/50 141/141 [==============================] - 1s 4ms/step - loss: 0.2825 - accuracy: 0.8881 - val_loss: 0.2647 - val_accuracy: 0.8930 Epoch 18/50 141/141 [==============================] - 1s 5ms/step - loss: 0.2825 - accuracy: 0.8874 - val_loss: 0.2641 - val_accuracy: 0.8940 Epoch 19/50 141/141 [==============================] - 1s 5ms/step - loss: 0.2803 - accuracy: 0.8877 - val_loss: 0.2637 - val_accuracy: 0.8944 Epoch 20/50 141/141 [==============================] - 1s 5ms/step - loss: 0.2800 - accuracy: 0.8881 - val_loss: 0.2631 - val_accuracy: 0.8948 Epoch 21/50 141/141 [==============================] - 1s 5ms/step - loss: 0.2792 - accuracy: 0.8884 - val_loss: 0.2631 - val_accuracy: 0.8948 Epoch 22/50 141/141 [==============================] - 0s 3ms/step - loss: 0.2781 - accuracy: 0.8885 - val_loss: 0.2618 - val_accuracy: 0.8953 Epoch 23/50 141/141 [==============================] - 0s 3ms/step - loss: 0.2779 - accuracy: 0.8885 - val_loss: 0.2610 - val_accuracy: 0.8950 Epoch 24/50 141/141 [==============================] - 1s 4ms/step - loss: 0.2750 - accuracy: 0.8893 - val_loss: 0.2606 - val_accuracy: 0.8954 Epoch 25/50 141/141 [==============================] - 1s 4ms/step - loss: 0.2749 - accuracy: 0.8891 - val_loss: 0.2596 - val_accuracy: 0.8955 Epoch 26/50 141/141 [==============================] - 1s 4ms/step - loss: 0.2742 - accuracy: 0.8894 - val_loss: 0.2592 - val_accuracy: 0.8955 Epoch 27/50 141/141 [==============================] - 1s 4ms/step - loss: 0.2737 - accuracy: 0.8893 - val_loss: 0.2586 - val_accuracy: 0.8964 Epoch 28/50 141/141 [==============================] - 1s 5ms/step - loss: 0.2736 - accuracy: 0.8892 - val_loss: 0.2582 - val_accuracy: 0.8967 Epoch 29/50 141/141 [==============================] - 1s 5ms/step - loss: 0.2730 - accuracy: 0.8900 - val_loss: 0.2580 - val_accuracy: 0.8970 Epoch 30/50 141/141 [==============================] - 1s 4ms/step - loss: 0.2725 - accuracy: 0.8907 - val_loss: 0.2573 - val_accuracy: 0.8974 Epoch 31/50 141/141 [==============================] - 1s 4ms/step - loss: 0.2718 - accuracy: 0.8898 - val_loss: 0.2568 - val_accuracy: 0.8966 Epoch 32/50 141/141 [==============================] - 1s 4ms/step - loss: 0.2725 - accuracy: 0.8894 - val_loss: 0.2568 - val_accuracy: 0.8971 Epoch 33/50 141/141 [==============================] - 1s 5ms/step - loss: 0.2711 - accuracy: 0.8902 - val_loss: 0.2557 - val_accuracy: 0.8976 Epoch 34/50 141/141 [==============================] - 1s 5ms/step - loss: 0.2699 - accuracy: 0.8905 - val_loss: 0.2557 - val_accuracy: 0.8976 Epoch 35/50 141/141 [==============================] - 1s 5ms/step - loss: 0.2700 - accuracy: 0.8907 - val_loss: 0.2557 - val_accuracy: 0.8977 Epoch 36/50 141/141 [==============================] - 1s 5ms/step - loss: 0.2696 - accuracy: 0.8909 - val_loss: 0.2546 - val_accuracy: 0.8976 Epoch 37/50 141/141 [==============================] - 1s 5ms/step - loss: 0.2696 - accuracy: 0.8909 - val_loss: 0.2546 - val_accuracy: 0.8979 Epoch 38/50 141/141 [==============================] - 1s 5ms/step - loss: 0.2691 - accuracy: 0.8906 - val_loss: 0.2544 - val_accuracy: 0.8979 Epoch 39/50 141/141 [==============================] - 1s 4ms/step - loss: 0.2690 - accuracy: 0.8917 - val_loss: 0.2546 - val_accuracy: 0.8986 Epoch 40/50 141/141 [==============================] - 1s 5ms/step - loss: 0.2679 - accuracy: 0.8918 - val_loss: 0.2539 - val_accuracy: 0.8983 Epoch 41/50 141/141 [==============================] - 1s 5ms/step - loss: 0.2673 - accuracy: 0.8922 - val_loss: 0.2529 - val_accuracy: 0.8988 Epoch 42/50 141/141 [==============================] - 1s 5ms/step - loss: 0.2684 - accuracy: 0.8918 - val_loss: 0.2529 - val_accuracy: 0.8984 Epoch 43/50 141/141 [==============================] - 1s 4ms/step - loss: 0.2674 - accuracy: 0.8928 - val_loss: 0.2532 - val_accuracy: 0.8985 Epoch 44/50 141/141 [==============================] - 1s 5ms/step - loss: 0.2662 - accuracy: 0.8931 - val_loss: 0.2526 - val_accuracy: 0.8988 Epoch 45/50 141/141 [==============================] - 1s 5ms/step - loss: 0.2674 - accuracy: 0.8918 - val_loss: 0.2521 - val_accuracy: 0.8994 Epoch 46/50 141/141 [==============================] - 1s 5ms/step - loss: 0.2653 - accuracy: 0.8931 - val_loss: 0.2529 - val_accuracy: 0.8987 Epoch 47/50 141/141 [==============================] - 1s 4ms/step - loss: 0.2661 - accuracy: 0.8929 - val_loss: 0.2517 - val_accuracy: 0.8988 Epoch 48/50 141/141 [==============================] - 0s 3ms/step - loss: 0.2666 - accuracy: 0.8923 - val_loss: 0.2513 - val_accuracy: 0.9000 Epoch 49/50 141/141 [==============================] - 0s 3ms/step - loss: 0.2654 - accuracy: 0.8930 - val_loss: 0.2517 - val_accuracy: 0.8996 Epoch 50/50 141/141 [==============================] - 1s 4ms/step - loss: 0.2654 - accuracy: 0.8936 - val_loss: 0.2514 - val_accuracy: 0.8997
plt.plot(history.history['loss'])
plt.title('Loss Change over Epoch')
plt.xlabel("Epoch")
plt.ylabel("Loss")
plt.yscale('log')
plt.show()
plt.plot(history.history['val_accuracy'])
plt.title('Validation Accuracy Change over Epoch')
plt.xlabel("Epoch")
plt.ylabel("Accuracy")
plt.show()
input_features_2 = ['lep1_pt','lep2_pt','fatjet_pt','fatjet_eta','fatjet_D2','Zll_mass','Zll_pt','MET','reco_zv_mass']
from tensorflow.keras.callbacks import EarlyStopping, ModelCheckpoint
input_data = sc.fit_transform(xtrain[input_features_2]) #redefine the input data
target = xtrain['isSignal']
# set random seed
Answer_to_all_questions = 42
# train - test split of dataset
train_data,test_data, train_target,test_target = model_selection.train_test_split(\
input_data,target,test_size =0.3,random_state = Answer_to_all_questions)
batchSize=500
N_epochs=100
num_inputs = 9
num_nodes = 50
extra_depth=5
num_outputs=1
model2 = my_model(num_inputs,num_outputs,num_nodes,extra_depth)
callbacks = [
# if we don’t have an increase of the accuracy for 10 epochs, terminate training.
EarlyStopping(verbose = True, patience = 10, monitor = 'val_accuracy'),
# Always make sure that we're saving the model weights with the best accuracy.
ModelCheckpoint('model.h5', monitor = 'val_accuracy', verbose = 0, save_best_only = True, mode = 'max')]
history2 = model2.fit(train_data, train_target, batch_size = batchSize, epochs = N_epochs, \
verbose = 1, validation_data = (test_data, test_target), callbacks = callbacks)
Epoch 1/100 141/141 [==============================] - 3s 9ms/step - loss: 0.3723 - accuracy: 0.8502 - val_loss: 0.1906 - val_accuracy: 0.9325 Epoch 2/100 141/141 [==============================] - 1s 6ms/step - loss: 0.1938 - accuracy: 0.9317 - val_loss: 0.1756 - val_accuracy: 0.9383 Epoch 3/100 141/141 [==============================] - 1s 7ms/step - loss: 0.1847 - accuracy: 0.9339 - val_loss: 0.1687 - val_accuracy: 0.9399 Epoch 4/100 141/141 [==============================] - 1s 8ms/step - loss: 0.1790 - accuracy: 0.9357 - val_loss: 0.1640 - val_accuracy: 0.9410 Epoch 5/100 141/141 [==============================] - 1s 7ms/step - loss: 0.1762 - accuracy: 0.9367 - val_loss: 0.1627 - val_accuracy: 0.9409 Epoch 6/100 141/141 [==============================] - 1s 8ms/step - loss: 0.1723 - accuracy: 0.9378 - val_loss: 0.1597 - val_accuracy: 0.9426 Epoch 7/100 141/141 [==============================] - 1s 8ms/step - loss: 0.1699 - accuracy: 0.9388 - val_loss: 0.1600 - val_accuracy: 0.9429 Epoch 8/100 141/141 [==============================] - 1s 8ms/step - loss: 0.1697 - accuracy: 0.9390 - val_loss: 0.1557 - val_accuracy: 0.9437 Epoch 9/100 141/141 [==============================] - 1s 8ms/step - loss: 0.1659 - accuracy: 0.9411 - val_loss: 0.1567 - val_accuracy: 0.9448 Epoch 10/100 141/141 [==============================] - 1s 7ms/step - loss: 0.1660 - accuracy: 0.9400 - val_loss: 0.1561 - val_accuracy: 0.9442 Epoch 11/100 141/141 [==============================] - 1s 8ms/step - loss: 0.1630 - accuracy: 0.9413 - val_loss: 0.1508 - val_accuracy: 0.9457 Epoch 12/100 141/141 [==============================] - 1s 7ms/step - loss: 0.1641 - accuracy: 0.9411 - val_loss: 0.1512 - val_accuracy: 0.9444 Epoch 13/100 141/141 [==============================] - 1s 7ms/step - loss: 0.1627 - accuracy: 0.9416 - val_loss: 0.1533 - val_accuracy: 0.9452 Epoch 14/100 141/141 [==============================] - 1s 7ms/step - loss: 0.1613 - accuracy: 0.9420 - val_loss: 0.1529 - val_accuracy: 0.9451 Epoch 15/100 141/141 [==============================] - 1s 8ms/step - loss: 0.1608 - accuracy: 0.9424 - val_loss: 0.1485 - val_accuracy: 0.9462 Epoch 16/100 141/141 [==============================] - 1s 6ms/step - loss: 0.1601 - accuracy: 0.9426 - val_loss: 0.1474 - val_accuracy: 0.9467 Epoch 17/100 141/141 [==============================] - 1s 7ms/step - loss: 0.1600 - accuracy: 0.9423 - val_loss: 0.1475 - val_accuracy: 0.9476 Epoch 18/100 141/141 [==============================] - 1s 6ms/step - loss: 0.1606 - accuracy: 0.9430 - val_loss: 0.1475 - val_accuracy: 0.9469 Epoch 19/100 141/141 [==============================] - 1s 7ms/step - loss: 0.1583 - accuracy: 0.9428 - val_loss: 0.1463 - val_accuracy: 0.9476 Epoch 20/100 141/141 [==============================] - 1s 7ms/step - loss: 0.1583 - accuracy: 0.9424 - val_loss: 0.1475 - val_accuracy: 0.9472 Epoch 21/100 141/141 [==============================] - 1s 7ms/step - loss: 0.1576 - accuracy: 0.9429 - val_loss: 0.1459 - val_accuracy: 0.9479 Epoch 22/100 141/141 [==============================] - 1s 8ms/step - loss: 0.1570 - accuracy: 0.9438 - val_loss: 0.1456 - val_accuracy: 0.9482 Epoch 23/100 141/141 [==============================] - 1s 7ms/step - loss: 0.1576 - accuracy: 0.9433 - val_loss: 0.1529 - val_accuracy: 0.9456 Epoch 24/100 141/141 [==============================] - 1s 7ms/step - loss: 0.1583 - accuracy: 0.9432 - val_loss: 0.1467 - val_accuracy: 0.9478 Epoch 25/100 141/141 [==============================] - 1s 7ms/step - loss: 0.1563 - accuracy: 0.9440 - val_loss: 0.1453 - val_accuracy: 0.9481 Epoch 26/100 141/141 [==============================] - 1s 7ms/step - loss: 0.1559 - accuracy: 0.9437 - val_loss: 0.1519 - val_accuracy: 0.9458 Epoch 27/100 141/141 [==============================] - 1s 7ms/step - loss: 0.1573 - accuracy: 0.9437 - val_loss: 0.1454 - val_accuracy: 0.9478 Epoch 28/100 141/141 [==============================] - 1s 7ms/step - loss: 0.1561 - accuracy: 0.9437 - val_loss: 0.1545 - val_accuracy: 0.9448 Epoch 29/100 141/141 [==============================] - 1s 7ms/step - loss: 0.1558 - accuracy: 0.9436 - val_loss: 0.1462 - val_accuracy: 0.9474 Epoch 30/100 141/141 [==============================] - 1s 7ms/step - loss: 0.1549 - accuracy: 0.9442 - val_loss: 0.1463 - val_accuracy: 0.9479 Epoch 31/100 141/141 [==============================] - 1s 7ms/step - loss: 0.1557 - accuracy: 0.9441 - val_loss: 0.1461 - val_accuracy: 0.9481 Epoch 32/100 141/141 [==============================] - 1s 6ms/step - loss: 0.1540 - accuracy: 0.9451 - val_loss: 0.1479 - val_accuracy: 0.9464 Epoch 32: early stopping
plt.plot(history2.history['loss'])
plt.title('Loss Change over Epoch')
plt.xlabel("Epoch")
plt.ylabel("Loss")
plt.yscale('log')
plt.show()
plt.plot(history2.history['val_accuracy'])
plt.title('Validation Accuracy Change over Epoch')
plt.xlabel("Epoch")
plt.ylabel("Accuracy")
plt.show()
print(model2.evaluate(test_data, test_target))
938/938 [==============================] - 2s 2ms/step - loss: 0.1479 - accuracy: 0.9464 [0.1478980928659439, 0.9464476108551025]
prediction = np.round(model2.predict(test_data),decimals=0)
print(prediction)
938/938 [==============================] - 2s 2ms/step [[0.] [1.] [0.] ... [0.] [1.] [0.]]
from sklearn.metrics import confusion_matrix
from sklearn.metrics import plot_confusion_matrix
from sklearn.metrics import ConfusionMatrixDisplay
cmat = confusion_matrix(test_target,prediction,normalize='true')
cmatplot = ConfusionMatrixDisplay(confusion_matrix=cmat,display_labels=['Background','Signal'])
cmatplot.plot()
plt.show()
# ROC Curve
from sklearn.metrics import roc_curve
# Create the roc curve
false_positive, true_positive, thresholds = roc_curve(test_target,model2.predict(test_data))
#plot the ROC curve
plt.plot(false_positive,true_positive)
plt.title("ROC Curve")
plt.xlabel('False Positive Rate')
plt.ylabel("True Positive Rate")
plt.show()
938/938 [==============================] - 2s 2ms/step
The ROC curve feature true positive rate on the Y axis, and false positive rate on the X axis. This means that the top left corner of the plot is the “ideal” point - a false positive rate of zero, and a true positive rate of one.