Skip to content
Snippets Groups Projects
Commit 6d8f1f77 authored by R.Schopper's avatar R.Schopper
Browse files

Merge branch 'personal_branch_ro_sc' into 'main'

Submission

See merge request !1
parents 452fa925 afbb12bf
Branches main
No related tags found
1 merge request!1Submission
Showing
with 2392 additions and 8 deletions
*.csv
*.txt
*.ipynb_checkpoints
\ No newline at end of file
*.ipynb_checkpoints
#ignore virtual environment folder
data/
\ No newline at end of file
Performing repeated nested cross validation with a data set of length 1521
Seed 1 - Split Variant 0 - Accuracy: 0.8235294117647058 - F1: 0.2702702702702703 - Recall: 0.29411764705882354 - MCC: 0.17137765534596305
Seed 1 - Split Variant 1 - Accuracy: 0.8881578947368421 - F1: 0.45161290322580644 - Recall: 0.4375 - MCC: 0.3896654075133036
Seed 1 - Split Variant 2 - Accuracy: 0.8881578947368421 - F1: 0.32 - Recall: 0.25 - MCC: 0.2772676883356913
Seed 1 - Split Variant 3 - Accuracy: 0.8947368421052632 - F1: 0.3333333333333333 - Recall: 0.25 - MCC: 0.30316953129541624
Seed 1 - Split Variant 4 - Accuracy: 0.8421052631578947 - F1: 0.14285714285714285 - Recall: 0.125 - MCC: 0.05857791861290061
Seed 1 - Split Variant 5 - Accuracy: 0.8355263157894737 - F1: 0.24242424242424243 - Recall: 0.23529411764705882 - MCC: 0.15035521186416326
Seed 1 - Split Variant 6 - Accuracy: 0.875 - F1: 0.3870967741935484 - Recall: 0.35294117647058826 - MCC: 0.3200847553716727
Seed 1 - Split Variant 7 - Accuracy: 0.881578947368421 - F1: 0.3076923076923077 - Recall: 0.23529411764705882 - MCC: 0.2647468358344461
Seed 1 - Split Variant 8 - Accuracy: 0.881578947368421 - F1: 0.35714285714285715 - Recall: 0.29411764705882354 - MCC: 0.303708801599408
Seed 1 - Split Variant 9 - Accuracy: 0.881578947368421 - F1: 0.35714285714285715 - Recall: 0.29411764705882354 - MCC: 0.303708801599408
Seed 3 - Split Variant 0 - Accuracy: 0.9019607843137255 - F1: 0.4 - Recall: 0.29411764705882354 - MCC: 0.38408534432354735
Seed 3 - Split Variant 1 - Accuracy: 0.8881578947368421 - F1: 0.2608695652173913 - Recall: 0.1875 - MCC: 0.23147067384109313
Seed 3 - Split Variant 2 - Accuracy: 0.881578947368421 - F1: 0.25 - Recall: 0.1875 - MCC: 0.2071658463852011
Seed 3 - Split Variant 3 - Accuracy: 0.868421052631579 - F1: 0.2857142857142857 - Recall: 0.25 - MCC: 0.21757512627648798
Seed 3 - Split Variant 4 - Accuracy: 0.881578947368421 - F1: 0.1 - Recall: 0.0625 - MCC: 0.07753403273475763
Seed 3 - Split Variant 5 - Accuracy: 0.8881578947368421 - F1: 0.2608695652173913 - Recall: 0.17647058823529413 - MCC: 0.24966626484129376
Seed 3 - Split Variant 6 - Accuracy: 0.8881578947368421 - F1: 0.2608695652173913 - Recall: 0.17647058823529413 - MCC: 0.24966626484129376
Seed 3 - Split Variant 7 - Accuracy: 0.8947368421052632 - F1: 0.5294117647058824 - Recall: 0.5294117647058824 - MCC: 0.47015250544662307
Seed 3 - Split Variant 8 - Accuracy: 0.8552631578947368 - F1: 0.35294117647058826 - Recall: 0.35294117647058826 - MCC: 0.27145969498910677
Seed 3 - Split Variant 9 - Accuracy: 0.8355263157894737 - F1: 0.2857142857142857 - Recall: 0.29411764705882354 - MCC: 0.19296380584816342
Seed 7 - Split Variant 0 - Accuracy: 0.8823529411764706 - F1: 0.3076923076923077 - Recall: 0.23529411764705882 - MCC: 0.2651650429449553
Seed 7 - Split Variant 1 - Accuracy: 0.8947368421052632 - F1: 0.2727272727272727 - Recall: 0.1875 - MCC: 0.2607480097276949
Seed 7 - Split Variant 2 - Accuracy: 0.8947368421052632 - F1: 0.5 - Recall: 0.5 - MCC: 0.4411764705882353
Seed 7 - Split Variant 3 - Accuracy: 0.8618421052631579 - F1: 0.3225806451612903 - Recall: 0.3125 - MCC: 0.24590535425596827
Seed 7 - Split Variant 4 - Accuracy: 0.9078947368421053 - F1: 0.2222222222222222 - Recall: 0.125 - MCC: 0.33665016461206926
Seed 7 - Split Variant 5 - Accuracy: 0.8486842105263158 - F1: 0.34285714285714286 - Recall: 0.35294117647058826 - MCC: 0.257568428070456
Seed 7 - Split Variant 6 - Accuracy: 0.868421052631579 - F1: 0.47368421052631576 - Recall: 0.5294117647058824 - MCC: 0.4023597368674538
Seed 7 - Split Variant 7 - Accuracy: 0.881578947368421 - F1: 0.18181818181818182 - Recall: 0.11764705882352941 - MCC: 0.168620005977641
Seed 7 - Split Variant 8 - Accuracy: 0.8618421052631579 - F1: 0.36363636363636365 - Recall: 0.35294117647058826 - MCC: 0.2863908797412633
Seed 7 - Split Variant 9 - Accuracy: 0.881578947368421 - F1: 0.4 - Recall: 0.35294117647058826 - MCC: 0.3393185685024823
Seed 9 - Split Variant 0 - Accuracy: 0.9019607843137255 - F1: 0.4 - Recall: 0.29411764705882354 - MCC: 0.38408534432354735
Seed 9 - Split Variant 1 - Accuracy: 0.9210526315789473 - F1: 0.5 - Recall: 0.375 - MCC: 0.4951769011158465
Seed 9 - Split Variant 2 - Accuracy: 0.881578947368421 - F1: 0.25 - Recall: 0.1875 - MCC: 0.2071658463852011
Seed 9 - Split Variant 3 - Accuracy: 0.9078947368421053 - F1: 0.4166666666666667 - Recall: 0.3125 - MCC: 0.3991732162056314
Seed 9 - Split Variant 4 - Accuracy: 0.9144736842105263 - F1: 0.43478260869565216 - Recall: 0.3125 - MCC: 0.43602615304950104
Seed 9 - Split Variant 5 - Accuracy: 0.8552631578947368 - F1: 0.15384615384615385 - Recall: 0.11764705882352941 - MCC: 0.08786103782637661
Seed 9 - Split Variant 6 - Accuracy: 0.8552631578947368 - F1: 0.21428571428571427 - Recall: 0.17647058823529413 - MCC: 0.1425788265798268
Seed 9 - Split Variant 7 - Accuracy: 0.8223684210526315 - F1: 0.18181818181818182 - Recall: 0.17647058823529413 - MCC: 0.0823373779256132
Seed 9 - Split Variant 8 - Accuracy: 0.8421052631578947 - F1: 0.2 - Recall: 0.17647058823529413 - MCC: 0.11539777655294262
Seed 9 - Split Variant 9 - Accuracy: 0.868421052631579 - F1: 0.4117647058823529 - Recall: 0.4117647058823529 - MCC: 0.33769063180827885
Seed 42 - Split Variant 0 - Accuracy: 0.9084967320261438 - F1: 0.36363636363636365 - Recall: 0.23529411764705882 - MCC: 0.4029033148052782
Seed 42 - Split Variant 1 - Accuracy: 0.875 - F1: 0.17391304347826086 - Recall: 0.125 - MCC: 0.12919293423688918
Seed 42 - Split Variant 2 - Accuracy: 0.875 - F1: 0.2962962962962963 - Recall: 0.25 - MCC: 0.23515185140414316
Seed 42 - Split Variant 3 - Accuracy: 0.8947368421052632 - F1: 0.1111111111111111 - Recall: 0.0625 - MCC: 0.14852213144650114
Seed 42 - Split Variant 4 - Accuracy: 0.9078947368421053 - F1: 0.5625 - Recall: 0.5625 - MCC: 0.5110294117647058
Seed 42 - Split Variant 5 - Accuracy: 0.9078947368421053 - F1: 0.5 - Recall: 0.4117647058823529 - MCC: 0.46483877661898915
Seed 42 - Split Variant 6 - Accuracy: 0.8421052631578947 - F1: 0.25 - Recall: 0.23529411764705882 - MCC: 0.16254643441594657
Seed 42 - Split Variant 7 - Accuracy: 0.9013157894736842 - F1: 0.34782608695652173 - Recall: 0.23529411764705882 - MCC: 0.35686759889744246
Seed 42 - Split Variant 8 - Accuracy: 0.8881578947368421 - F1: 0.41379310344827586 - Recall: 0.35294117647058826 - MCC: 0.3605680754977631
Seed 42 - Split Variant 9 - Accuracy: 0.8092105263157895 - F1: 0.2564102564102564 - Recall: 0.29411764705882354 - MCC: 0.15066519678920853
Repeated Nested Cross-Validation Accuracy: 0.8767870657034743
File added
import numpy as np
import math
# Output_RNCV2
file_name = 'Output_RNCV1_good.txt'
def avg(file, metric):
scores = []
with open(file) as f:
lines = f.readlines()
for line in lines:
sub_str = line.split()
for idx, string in enumerate(sub_str):
if string == metric+':':
if math.isnan(float(sub_str[idx+1])):
continue
else:
scores.append(float(sub_str[idx+1]))
#print(scores)
return np.mean(scores)
metric_list = ['Accuracy', 'F1', 'Recall', 'MCC']
metric_dict = {}
for metric in metric_list:
mean = avg(file_name, metric)
metric_dict[metric] = mean
print(metric_dict)
\ No newline at end of file
Searching for best clf in {'C': [0.1, 1, 10, 100], 'kernel': ['linear', 'poly', 'rbf', 'sigmoid'], 'gamma': ['auto', 'scale']} with 1521 datapoints with following features: ['TotalDwellTime', 'GazePointsAOI', 'Instances', 'AvTime', 'AvPoints', 'StDtime', 'StDpoints']
Performing Grid Search...
Fitting 5 folds for each of 32 candidates, totalling 160 fits
Seed 7 - Split Variant 0 - Test m-Accuracy: 0.8104575163398693
Performing fit of best model with params: C - 10; Kernel - rbf; Gamma - auto on full data set...
Seed 7 - Split Variant 0 - Accuracy: 0.8104575163398693 - F1: 0.8929889298892989 - Recall: 0.9918032786885246 - MCC: 0.22313151537444834
Searching for best clf in {'C': [0.1, 1, 10, 100], 'kernel': ['linear', 'poly', 'rbf', 'sigmoid'], 'gamma': ['auto', 'scale']} with 1521 datapoints with following features: ['TotalDwellTime', 'GazePointsAOI', 'Instances', 'AvTime', 'AvPoints', 'StDtime', 'StDpoints']
Performing Grid Search...
Fitting 5 folds for each of 32 candidates, totalling 160 fits
Seed 7 - Split Variant 0 - Test m-Accuracy: 0.8169934640522876
Performing fit of best model with params: C - 100; Kernel - poly; Gamma - scale on full data set...
Seed 7 - Split Variant 0 - Accuracy: 0.8169934640522876 - F1: 0.2222222222222222 - Recall: 0.26666666666666666 - MCC: 0.12398657510978121
Searching for best clf in {'C': [0.1, 1, 10, 100], 'kernel': ['linear', 'poly', 'rbf', 'sigmoid'], 'gamma': ['auto', 'scale']} with 1521 datapoints with following features: ['TotalDwellTime', 'GazePointsAOI', 'Instances', 'AvTime', 'AvPoints', 'StDtime', 'StDpoints']
Performing Grid Search...
Fitting 5 folds for each of 32 candidates, totalling 160 fits
Seed 7 - Split Variant 0 - Test m-Accuracy: 0.8823529411764706
Performing fit of best model with params: C - 0.1; Kernel - poly; Gamma - scale on full data set...
Seed 7 - Split Variant 0 - Accuracy: 0.8823529411764706 - F1: 0.3076923076923077 - Recall: 0.23529411764705882 - MCC: 0.2651650429449553
Total Acc: 0.8139381985535832 - Bad: 0.8132807363576594 - Good: 0.9322813938198553 - Expert: 0.9441157133464826
percent of bad labels: 0.7975016436554898
\ No newline at end of file
Performing repeated nested cross validation with a data set of length 1521
Seed 1 - Split Variant 0 - Accuracy: 0.8104575163398693 - F1: 0.8937728937728938 - Recall: 1.0 - MCC: 0.2283103462066562
Seed 1 - Split Variant 1 - Accuracy: 0.8157894736842105 - F1: 0.8955223880597015 - Recall: 0.9836065573770492 - MCC: 0.23902931848612277
Seed 1 - Split Variant 2 - Accuracy: 0.8157894736842105 - F1: 0.8962962962962963 - Recall: 0.9918032786885246 - MCC: 0.2282642406947996
Seed 1 - Split Variant 3 - Accuracy: 0.7960526315789473 - F1: 0.8847583643122676 - Recall: 0.9834710743801653 - MCC: 0.12079198541403256
Seed 1 - Split Variant 4 - Accuracy: 0.8157894736842105 - F1: 0.8947368421052632 - Recall: 0.9834710743801653 - MCC: 0.2782872289962015
Seed 1 - Split Variant 5 - Accuracy: 0.8223684210526315 - F1: 0.8996282527881041 - Recall: 1.0 - MCC: 0.3247962274466209
Seed 1 - Split Variant 6 - Accuracy: 0.7894736842105263 - F1: 0.8805970149253731 - Recall: 0.9752066115702479 - MCC: 0.08973648762112346
Seed 1 - Split Variant 7 - Accuracy: 0.7960526315789473 - F1: 0.8847583643122676 - Recall: 0.9834710743801653 - MCC: 0.12079198541403256
Seed 1 - Split Variant 8 - Accuracy: 0.8092105263157895 - F1: 0.8929889298892989 - Recall: 1.0 - MCC: 0.22812937284069357
Seed 1 - Split Variant 9 - Accuracy: 0.8026315789473685 - F1: 0.8872180451127819 - Recall: 0.9752066115702479 - MCC: 0.20038730485730166
Seed 3 - Split Variant 0 - Accuracy: 0.8169934640522876 - F1: 0.8962962962962963 - Recall: 0.9918032786885246 - MCC: 0.27317413586995526
Seed 3 - Split Variant 1 - Accuracy: 0.8157894736842105 - F1: 0.8970588235294118 - Recall: 1.0 - MCC: 0.2328566559543064
Seed 3 - Split Variant 2 - Accuracy: 0.8157894736842105 - F1: 0.8970588235294118 - Recall: 1.0 - MCC: 0.2328566559543064
Seed 3 - Split Variant 3 - Accuracy: 0.7894736842105263 - F1: 0.8814814814814815 - Recall: 0.9834710743801653 - MCC: 0.04556430161200848
Seed 3 - Split Variant 4 - Accuracy: 0.8026315789473685 - F1: 0.8888888888888888 - Recall: 0.9917355371900827 - MCC: 0.16295029898531846
Seed 3 - Split Variant 5 - Accuracy: 0.8092105263157895 - F1: 0.8913857677902621 - Recall: 0.9834710743801653 - MCC: 0.2328020851178598
Seed 3 - Split Variant 6 - Accuracy: 0.8026315789473685 - F1: 0.8880597014925373 - Recall: 0.9834710743801653 - MCC: 0.18127975016079304
Seed 3 - Split Variant 7 - Accuracy: 0.7828947368421053 - F1: 0.8764044943820225 - Recall: 0.9669421487603306 - MCC: 0.06509631763959113
Seed 3 - Split Variant 8 - Accuracy: 0.8289473684210527 - F1: 0.9022556390977443 - Recall: 0.9917355371900827 - MCC: 0.3561871531351014
Seed 3 - Split Variant 9 - Accuracy: 0.8223684210526315 - F1: 0.8996282527881041 - Recall: 1.0 - MCC: 0.3247962274466209
Seed 7 - Split Variant 0 - Accuracy: 0.8104575163398693 - F1: 0.8929889298892989 - Recall: 0.9918032786885246 - MCC: 0.22313151537444834
Seed 7 - Split Variant 1 - Accuracy: 0.7960526315789473 - F1: 0.8856088560885609 - Recall: 0.9836065573770492 - MCC: 0.048472719420948565
Seed 7 - Split Variant 2 - Accuracy: 0.7894736842105263 - F1: 0.8787878787878788 - Recall: 0.9508196721311475 - MCC: 0.1351033138402079
Seed 7 - Split Variant 3 - Accuracy: 0.8026315789473685 - F1: 0.8897058823529411 - Recall: 1.0 - MCC: 0.16077679411584753
Seed 7 - Split Variant 4 - Accuracy: 0.8026315789473685 - F1: 0.8872180451127819 - Recall: 0.9752066115702479 - MCC: 0.20038730485730166
Seed 7 - Split Variant 5 - Accuracy: 0.7763157894736842 - F1: 0.8721804511278195 - Recall: 0.9586776859504132 - MCC: 0.0445874565795019
Seed 7 - Split Variant 6 - Accuracy: 0.8026315789473685 - F1: 0.8888888888888888 - Recall: 0.9917355371900827 - MCC: 0.16295029898531846
Seed 7 - Split Variant 7 - Accuracy: 0.7894736842105263 - F1: 0.8814814814814815 - Recall: 0.9834710743801653 - MCC: 0.04556430161200848
Seed 7 - Split Variant 8 - Accuracy: 0.8092105263157895 - F1: 0.8913857677902621 - Recall: 0.9834710743801653 - MCC: 0.2328020851178598
Seed 7 - Split Variant 9 - Accuracy: 0.8223684210526315 - F1: 0.8996282527881041 - Recall: 1.0 - MCC: 0.3247962274466209
Seed 9 - Split Variant 0 - Accuracy: 0.8104575163398693 - F1: 0.8937728937728938 - Recall: 1.0 - MCC: 0.2283103462066562
Seed 9 - Split Variant 1 - Accuracy: 0.8223684210526315 - F1: 0.9003690036900369 - Recall: 1.0 - MCC: 0.28614540819463186
Seed 9 - Split Variant 2 - Accuracy: 0.8157894736842105 - F1: 0.8962962962962963 - Recall: 0.9918032786885246 - MCC: 0.2282642406947996
Seed 9 - Split Variant 3 - Accuracy: 0.7894736842105263 - F1: 0.8796992481203008 - Recall: 0.9669421487603306 - MCC: 0.12248738071840178
Seed 9 - Split Variant 4 - Accuracy: 0.8157894736842105 - F1: 0.8947368421052632 - Recall: 0.9834710743801653 - MCC: 0.2782872289962015
Seed 9 - Split Variant 5 - Accuracy: 0.7894736842105263 - F1: 0.8805970149253731 - Recall: 0.9752066115702479 - MCC: 0.08973648762112346
Seed 9 - Split Variant 6 - Accuracy: 0.7828947368421053 - F1: 0.8782287822878229 - Recall: 0.9834710743801653 - MCC: -0.058446368248442154
Seed 9 - Split Variant 7 - Accuracy: 0.8092105263157895 - F1: 0.8921933085501859 - Recall: 0.9917355371900827 - MCC: 0.22279410643032674
Seed 9 - Split Variant 8 - Accuracy: 0.8223684210526315 - F1: 0.898876404494382 - Recall: 0.9917355371900827 - MCC: 0.3166549688569941
Seed 9 - Split Variant 9 - Accuracy: 0.7960526315789473 - F1: 0.8847583643122676 - Recall: 0.9834710743801653 - MCC: 0.12079198541403256
Seed 42 - Split Variant 0 - Accuracy: 0.8169934640522876 - F1: 0.8970588235294118 - Recall: 1.0 - MCC: 0.28055245038914706
Seed 42 - Split Variant 1 - Accuracy: 0.8157894736842105 - F1: 0.8962962962962963 - Recall: 0.9918032786885246 - MCC: 0.2282642406947996
Seed 42 - Split Variant 2 - Accuracy: 0.8026315789473685 - F1: 0.8897058823529411 - Recall: 0.9918032786885246 - MCC: 0.08779841126145979
Seed 42 - Split Variant 3 - Accuracy: 0.8223684210526315 - F1: 0.898876404494382 - Recall: 0.9917355371900827 - MCC: 0.3166549688569941
Seed 42 - Split Variant 4 - Accuracy: 0.8092105263157895 - F1: 0.8929889298892989 - Recall: 1.0 - MCC: 0.22812937284069357
Seed 42 - Split Variant 5 - Accuracy: 0.7828947368421053 - F1: 0.8782287822878229 - Recall: 0.9834710743801653 - MCC: -0.058446368248442154
Seed 42 - Split Variant 6 - Accuracy: 0.7828947368421053 - F1: 0.8754716981132076 - Recall: 0.9586776859504132 - MCC: 0.10006070612136221
Seed 42 - Split Variant 7 - Accuracy: 0.8157894736842105 - F1: 0.8955223880597015 - Recall: 0.9917355371900827 - MCC: 0.2728230127004626
Seed 42 - Split Variant 8 - Accuracy: 0.8223684210526315 - F1: 0.898876404494382 - Recall: 0.9917355371900827 - MCC: 0.3166549688569941
Seed 42 - Split Variant 9 - Accuracy: 0.8092105263157895 - F1: 0.8921933085501859 - Recall: 0.9917355371900827 - MCC: 0.22279410643032674
Repeated Nested Cross-Validation Accuracy: 0.8059124527003784
F1 score for field TotalDwellTime: 0.8884826325411335
F1 score for field GazePointsAOI: 0.8884826325411335
F1 score for field Instances: 0.8827838827838828
F1 score for field AvTime: 0.8905109489051095
F1 score for field AvPoints: 0.8905109489051095
F1 score for field StDtime: 0.8888888888888888
F1 score for field StDpoints: 0.8888888888888888
Selected feature: AvTime with index 3
F1 score for field TotalDwellTime: 0.8782287822878229
F1 score for field GazePointsAOI: 0.8782287822878229
F1 score for field Instances: 0.8798521256931608
F1 score for field AvPoints: 0.8823529411764706
F1 score for field StDtime: 0.8905109489051095
F1 score for field StDpoints: 0.8905109489051095
Selected feature: StDtime with index 4
F1 score for field TotalDwellTime: 0.8782287822878229
F1 score for field GazePointsAOI: 0.8782287822878229
F1 score for field Instances: 0.8798521256931608
F1 score for field AvPoints: 0.8823529411764706
F1 score for field StDpoints: 0.8905109489051095
Selected feature: StDpoints with index 4
F1 score for field TotalDwellTime: 0.8802946593001841
F1 score for field GazePointsAOI: 0.8802946593001841
F1 score for field Instances: 0.8802946593001841
F1 score for field AvPoints: 0.8802946593001841
Selected feature: TotalDwellTime with index 0
F1 score for field GazePointsAOI: 0.8761552680221811
F1 score for field Instances: 0.8719851576994434
F1 score for field AvPoints: 0.8782287822878229
Selected feature: AvPoints with index 2
F1 score for field GazePointsAOI: 0.8761552680221811
F1 score for field Instances: 0.87569573283859
Selected feature: GazePointsAOI with index 0
F1 score for field Instances: 0.8736059479553904
Selected feature: Instances with index 0
picked_feats_record = ['AvTime', 'StDtime', 'StDpoints', 'TotalDwellTime', 'AvPoints', 'GazePointsAOI', 'Instances']
f1_score_feature_selection = [0.8905109489051095, 0.8905109489051095, 0.8905109489051095, 0.8782287822878229, 0.8802946593001841, 0.8761552680221811, 0.8736059479553904]
\ No newline at end of file
Searching for best clf in {'C': [0.1, 1, 10, 100], 'kernel': ['linear', 'poly', 'rbf', 'sigmoid'], 'gamma': ['auto', 'scale']} with 1521 datapoints with following features: ['Instances', 'AvTime', 'StDtime']
Performing Grid Search...
Fitting 5 folds for each of 32 candidates, totalling 160 fits
Seed 7 - Split Variant 0 - Test m-Accuracy: 0.803921568627451
Performing fit of best model with params: C - 100; Kernel - sigmoid; Gamma - auto on full data set...
Seed 7 - Split Variant 0 - Accuracy: 0.803921568627451 - F1: 0.8897058823529411 - Recall: 0.9918032786885246 - MCC: 0.16327232768548725
Probability of null hypothesis: 0.001
\ No newline at end of file
Performing repeated nested cross validation with a data set of length 1521
Seed 1 - Split Variant 0 - Accuracy: 0.8235294117647058 - F1: 0.2702702702702703 - Recall: 0.29411764705882354 - MCC: 0.17137765534596305
Seed 1 - Split Variant 1 - Accuracy: 0.8881578947368421 - F1: 0.45161290322580644 - Recall: 0.4375 - MCC: 0.3896654075133036
Seed 1 - Split Variant 2 - Accuracy: 0.8881578947368421 - F1: 0.32 - Recall: 0.25 - MCC: 0.2772676883356913
Seed 1 - Split Variant 3 - Accuracy: 0.8947368421052632 - F1: 0.3333333333333333 - Recall: 0.25 - MCC: 0.30316953129541624
Seed 1 - Split Variant 4 - Accuracy: 0.8421052631578947 - F1: 0.14285714285714285 - Recall: 0.125 - MCC: 0.05857791861290061
Seed 1 - Split Variant 5 - Accuracy: 0.8355263157894737 - F1: 0.24242424242424243 - Recall: 0.23529411764705882 - MCC: 0.15035521186416326
Seed 1 - Split Variant 6 - Accuracy: 0.875 - F1: 0.3870967741935484 - Recall: 0.35294117647058826 - MCC: 0.3200847553716727
Seed 1 - Split Variant 7 - Accuracy: 0.881578947368421 - F1: 0.3076923076923077 - Recall: 0.23529411764705882 - MCC: 0.2647468358344461
Seed 1 - Split Variant 8 - Accuracy: 0.881578947368421 - F1: 0.35714285714285715 - Recall: 0.29411764705882354 - MCC: 0.303708801599408
Seed 1 - Split Variant 9 - Accuracy: 0.881578947368421 - F1: 0.35714285714285715 - Recall: 0.29411764705882354 - MCC: 0.303708801599408
Seed 3 - Split Variant 0 - Accuracy: 0.9019607843137255 - F1: 0.4 - Recall: 0.29411764705882354 - MCC: 0.38408534432354735
Seed 3 - Split Variant 1 - Accuracy: 0.8881578947368421 - F1: 0.2608695652173913 - Recall: 0.1875 - MCC: 0.23147067384109313
Seed 3 - Split Variant 2 - Accuracy: 0.881578947368421 - F1: 0.25 - Recall: 0.1875 - MCC: 0.2071658463852011
Seed 3 - Split Variant 3 - Accuracy: 0.868421052631579 - F1: 0.2857142857142857 - Recall: 0.25 - MCC: 0.21757512627648798
Seed 3 - Split Variant 4 - Accuracy: 0.881578947368421 - F1: 0.1 - Recall: 0.0625 - MCC: 0.07753403273475763
Seed 3 - Split Variant 5 - Accuracy: 0.8881578947368421 - F1: 0.2608695652173913 - Recall: 0.17647058823529413 - MCC: 0.24966626484129376
Seed 3 - Split Variant 6 - Accuracy: 0.8881578947368421 - F1: 0.2608695652173913 - Recall: 0.17647058823529413 - MCC: 0.24966626484129376
Seed 3 - Split Variant 7 - Accuracy: 0.8947368421052632 - F1: 0.5294117647058824 - Recall: 0.5294117647058824 - MCC: 0.47015250544662307
Seed 3 - Split Variant 8 - Accuracy: 0.8552631578947368 - F1: 0.35294117647058826 - Recall: 0.35294117647058826 - MCC: 0.27145969498910677
Seed 3 - Split Variant 9 - Accuracy: 0.8355263157894737 - F1: 0.2857142857142857 - Recall: 0.29411764705882354 - MCC: 0.19296380584816342
Seed 7 - Split Variant 0 - Accuracy: 0.8823529411764706 - F1: 0.3076923076923077 - Recall: 0.23529411764705882 - MCC: 0.2651650429449553
Seed 7 - Split Variant 1 - Accuracy: 0.8947368421052632 - F1: 0.2727272727272727 - Recall: 0.1875 - MCC: 0.2607480097276949
Seed 7 - Split Variant 2 - Accuracy: 0.8947368421052632 - F1: 0.5 - Recall: 0.5 - MCC: 0.4411764705882353
Seed 7 - Split Variant 3 - Accuracy: 0.8618421052631579 - F1: 0.3225806451612903 - Recall: 0.3125 - MCC: 0.24590535425596827
Seed 7 - Split Variant 4 - Accuracy: 0.9078947368421053 - F1: 0.2222222222222222 - Recall: 0.125 - MCC: 0.33665016461206926
Seed 7 - Split Variant 5 - Accuracy: 0.8486842105263158 - F1: 0.34285714285714286 - Recall: 0.35294117647058826 - MCC: 0.257568428070456
Seed 7 - Split Variant 6 - Accuracy: 0.868421052631579 - F1: 0.47368421052631576 - Recall: 0.5294117647058824 - MCC: 0.4023597368674538
Seed 7 - Split Variant 7 - Accuracy: 0.881578947368421 - F1: 0.18181818181818182 - Recall: 0.11764705882352941 - MCC: 0.168620005977641
Seed 7 - Split Variant 8 - Accuracy: 0.8618421052631579 - F1: 0.36363636363636365 - Recall: 0.35294117647058826 - MCC: 0.2863908797412633
Seed 7 - Split Variant 9 - Accuracy: 0.881578947368421 - F1: 0.4 - Recall: 0.35294117647058826 - MCC: 0.3393185685024823
Seed 9 - Split Variant 0 - Accuracy: 0.9019607843137255 - F1: 0.4 - Recall: 0.29411764705882354 - MCC: 0.38408534432354735
Seed 9 - Split Variant 1 - Accuracy: 0.9210526315789473 - F1: 0.5 - Recall: 0.375 - MCC: 0.4951769011158465
Seed 9 - Split Variant 2 - Accuracy: 0.881578947368421 - F1: 0.25 - Recall: 0.1875 - MCC: 0.2071658463852011
Seed 9 - Split Variant 3 - Accuracy: 0.9078947368421053 - F1: 0.4166666666666667 - Recall: 0.3125 - MCC: 0.3991732162056314
Seed 9 - Split Variant 4 - Accuracy: 0.9144736842105263 - F1: 0.43478260869565216 - Recall: 0.3125 - MCC: 0.43602615304950104
Seed 9 - Split Variant 5 - Accuracy: 0.8552631578947368 - F1: 0.15384615384615385 - Recall: 0.11764705882352941 - MCC: 0.08786103782637661
Seed 9 - Split Variant 6 - Accuracy: 0.8552631578947368 - F1: 0.21428571428571427 - Recall: 0.17647058823529413 - MCC: 0.1425788265798268
Seed 9 - Split Variant 7 - Accuracy: 0.8223684210526315 - F1: 0.18181818181818182 - Recall: 0.17647058823529413 - MCC: 0.0823373779256132
Seed 9 - Split Variant 8 - Accuracy: 0.8421052631578947 - F1: 0.2 - Recall: 0.17647058823529413 - MCC: 0.11539777655294262
Seed 9 - Split Variant 9 - Accuracy: 0.868421052631579 - F1: 0.4117647058823529 - Recall: 0.4117647058823529 - MCC: 0.33769063180827885
Seed 42 - Split Variant 0 - Accuracy: 0.9084967320261438 - F1: 0.36363636363636365 - Recall: 0.23529411764705882 - MCC: 0.4029033148052782
Seed 42 - Split Variant 1 - Accuracy: 0.875 - F1: 0.17391304347826086 - Recall: 0.125 - MCC: 0.12919293423688918
Seed 42 - Split Variant 2 - Accuracy: 0.875 - F1: 0.2962962962962963 - Recall: 0.25 - MCC: 0.23515185140414316
Seed 42 - Split Variant 3 - Accuracy: 0.8947368421052632 - F1: 0.1111111111111111 - Recall: 0.0625 - MCC: 0.14852213144650114
Seed 42 - Split Variant 4 - Accuracy: 0.9078947368421053 - F1: 0.5625 - Recall: 0.5625 - MCC: 0.5110294117647058
Seed 42 - Split Variant 5 - Accuracy: 0.9078947368421053 - F1: 0.5 - Recall: 0.4117647058823529 - MCC: 0.46483877661898915
Seed 42 - Split Variant 6 - Accuracy: 0.8421052631578947 - F1: 0.25 - Recall: 0.23529411764705882 - MCC: 0.16254643441594657
Seed 42 - Split Variant 7 - Accuracy: 0.9013157894736842 - F1: 0.34782608695652173 - Recall: 0.23529411764705882 - MCC: 0.35686759889744246
Seed 42 - Split Variant 8 - Accuracy: 0.8881578947368421 - F1: 0.41379310344827586 - Recall: 0.35294117647058826 - MCC: 0.3605680754977631
Seed 42 - Split Variant 9 - Accuracy: 0.8092105263157895 - F1: 0.2564102564102564 - Recall: 0.29411764705882354 - MCC: 0.15066519678920853
Repeated Nested Cross-Validation Accuracy: 0.8767870657034743
F1 score for field TotalDwellTime: 0.0
F1 score for field GazePointsAOI: 0.0
F1 score for field Instances: 0.10256410256410256
F1 score for field AvTime: 0.0
F1 score for field AvPoints: 0.0
F1 score for field StDtime: 0.0
F1 score for field StDpoints: 0.0
Selected feature: Instances with index 2
F1 score for field TotalDwellTime: 0.22727272727272727
F1 score for field GazePointsAOI: 0.22727272727272727
F1 score for field AvTime: 0.2727272727272727
F1 score for field AvPoints: 0.2727272727272727
F1 score for field StDtime: 0.10526315789473684
F1 score for field StDpoints: 0.05405405405405406
Selected feature: AvTime with index 2
F1 score for field TotalDwellTime: 0.2608695652173913
F1 score for field GazePointsAOI: 0.2608695652173913
F1 score for field AvPoints: 0.2608695652173913
F1 score for field StDtime: 0.27906976744186046
F1 score for field StDpoints: 0.23809523809523808
Selected feature: StDtime with index 3
F1 score for field TotalDwellTime: 0.3111111111111111
F1 score for field GazePointsAOI: 0.3111111111111111
F1 score for field AvPoints: 0.2608695652173913
F1 score for field StDpoints: 0.24390243902439024
Selected feature: TotalDwellTime with index 0
F1 score for field GazePointsAOI: 0.2978723404255319
F1 score for field AvPoints: 0.2978723404255319
F1 score for field StDpoints: 0.2727272727272727
Selected feature: GazePointsAOI with index 0
F1 score for field AvPoints: 0.30434782608695654
F1 score for field StDpoints: 0.2727272727272727
Selected feature: AvPoints with index 0
F1 score for field StDpoints: 0.26666666666666666
Selected feature: StDpoints with index 0
picked_feats_record = ['Instances', 'AvTime', 'StDtime', 'TotalDwellTime', 'GazePointsAOI', 'AvPoints', 'StDpoints']
f1_score_feature_selection = [0.10256410256410256, 0.2727272727272727, 0.27906976744186046, 0.3111111111111111, 0.2978723404255319, 0.30434782608695654, 0.26666666666666666]
\ No newline at end of file
Searching for best clf in {'C': [0.1, 1, 10, 100], 'kernel': ['linear', 'poly', 'rbf', 'sigmoid'], 'gamma': ['auto', 'scale']} with 1521 datapoints with following features: ['Instances', 'AvTime', 'StDtime', 'TotalDwellTime']
Performing Grid Search...
Fitting 5 folds for each of 32 candidates, totalling 160 fits
Seed 7 - Split Variant 0 - Test m-Accuracy: 0.9019607843137255
Performing fit of best model with params: C - 0.1; Kernel - linear; Gamma - auto on full data set...
Seed 7 - Split Variant 0 - n-Accuracy: 0.9019607843137255
Probability of null hypothesis: 0.001
\ No newline at end of file
Performing repeated nested cross validation with a data set of length 1521
Seed 1 - Split Variant 0 - Accuracy: 0.9019607843137255 - F1: 0.9484536082474226 - Recall: 1.0 - MCC: nan
Seed 1 - Split Variant 1 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 1 - Split Variant 2 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 1 - Split Variant 3 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 1 - Split Variant 4 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 1 - Split Variant 5 - Accuracy: 0.9144736842105263 - F1: 0.9550173010380623 - Recall: 1.0 - MCC: 0.2554977333933865
Seed 1 - Split Variant 6 - Accuracy: 0.9144736842105263 - F1: 0.9550173010380623 - Recall: 1.0 - MCC: 0.2554977333933865
Seed 1 - Split Variant 7 - Accuracy: 0.9013157894736842 - F1: 0.9480968858131488 - Recall: 0.9927536231884058 - MCC: -0.02592005990947399
Seed 1 - Split Variant 8 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 1 - Split Variant 9 - Accuracy: 0.9013157894736842 - F1: 0.9480968858131488 - Recall: 1.0 - MCC: nan
Seed 3 - Split Variant 0 - Accuracy: 0.9019607843137255 - F1: 0.9484536082474226 - Recall: 1.0 - MCC: nan
Seed 3 - Split Variant 1 - Accuracy: 0.9144736842105263 - F1: 0.9550173010380623 - Recall: 1.0 - MCC: 0.2554977333933865
Seed 3 - Split Variant 2 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 3 - Split Variant 3 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 3 - Split Variant 4 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 3 - Split Variant 5 - Accuracy: 0.9013157894736842 - F1: 0.9480968858131488 - Recall: 0.9927536231884058 - MCC: -0.02592005990947399
Seed 3 - Split Variant 6 - Accuracy: 0.9210526315789473 - F1: 0.9583333333333334 - Recall: 1.0 - MCC: 0.3625307868699863
Seed 3 - Split Variant 7 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 3 - Split Variant 8 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 3 - Split Variant 9 - Accuracy: 0.9013157894736842 - F1: 0.9480968858131488 - Recall: 1.0 - MCC: nan
Seed 7 - Split Variant 0 - Accuracy: 0.9019607843137255 - F1: 0.9484536082474226 - Recall: 1.0 - MCC: nan
Seed 7 - Split Variant 1 - Accuracy: 0.9144736842105263 - F1: 0.9550173010380623 - Recall: 1.0 - MCC: 0.2554977333933865
Seed 7 - Split Variant 2 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 7 - Split Variant 3 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 7 - Split Variant 4 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 7 - Split Variant 5 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 7 - Split Variant 6 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 7 - Split Variant 7 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 7 - Split Variant 8 - Accuracy: 0.9013157894736842 - F1: 0.9480968858131488 - Recall: 0.9927536231884058 - MCC: -0.02592005990947399
Seed 7 - Split Variant 9 - Accuracy: 0.9013157894736842 - F1: 0.9480968858131488 - Recall: 1.0 - MCC: nan
Seed 9 - Split Variant 0 - Accuracy: 0.9019607843137255 - F1: 0.9484536082474226 - Recall: 1.0 - MCC: nan
Seed 9 - Split Variant 1 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 9 - Split Variant 2 - Accuracy: 0.9144736842105263 - F1: 0.9550173010380623 - Recall: 1.0 - MCC: 0.2554977333933865
Seed 9 - Split Variant 3 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 9 - Split Variant 4 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 9 - Split Variant 5 - Accuracy: 0.9013157894736842 - F1: 0.9480968858131488 - Recall: 0.9927536231884058 - MCC: -0.02592005990947399
Seed 9 - Split Variant 6 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 9 - Split Variant 7 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 9 - Split Variant 8 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 9 - Split Variant 9 - Accuracy: 0.9013157894736842 - F1: 0.9480968858131488 - Recall: 1.0 - MCC: nan
Seed 42 - Split Variant 0 - Accuracy: 0.9019607843137255 - F1: 0.9484536082474226 - Recall: 1.0 - MCC: nan
Seed 42 - Split Variant 1 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 42 - Split Variant 2 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 42 - Split Variant 3 - Accuracy: 0.9144736842105263 - F1: 0.9550173010380623 - Recall: 1.0 - MCC: 0.2554977333933865
Seed 42 - Split Variant 4 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 42 - Split Variant 5 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 42 - Split Variant 6 - Accuracy: 0.9144736842105263 - F1: 0.9550173010380623 - Recall: 1.0 - MCC: 0.2554977333933865
Seed 42 - Split Variant 7 - Accuracy: 0.9013157894736842 - F1: 0.9477351916376306 - Recall: 0.9855072463768116 - MCC: 0.11836828697545493
Seed 42 - Split Variant 8 - Accuracy: 0.9078947368421053 - F1: 0.9517241379310345 - Recall: 1.0 - MCC: nan
Seed 42 - Split Variant 9 - Accuracy: 0.9013157894736842 - F1: 0.9480968858131488 - Recall: 1.0 - MCC: nan
Repeated Nested Cross-Validation Accuracy: 0.907169762641899
F1 score for field TotalDwellTime: 0.9428076256499134
F1 score for field GazePointsAOI: 0.9428076256499134
F1 score for field Instances: 0.9428076256499134
F1 score for field AvTime: 0.9428076256499134
F1 score for field AvPoints: 0.9428076256499134
F1 score for field StDtime: 0.9428076256499134
F1 score for field StDpoints: 0.9428076256499134
Selected feature: TotalDwellTime with index 0
F1 score for field GazePointsAOI: 0.9428076256499134
F1 score for field Instances: 0.9428076256499134
F1 score for field AvTime: 0.9428076256499134
F1 score for field AvPoints: 0.9428076256499134
F1 score for field StDtime: 0.9428076256499134
F1 score for field StDpoints: 0.9428076256499134
Selected feature: GazePointsAOI with index 0
F1 score for field Instances: 0.9428076256499134
F1 score for field AvTime: 0.9428076256499134
F1 score for field AvPoints: 0.9428076256499134
F1 score for field StDtime: 0.9428076256499134
F1 score for field StDpoints: 0.9428076256499134
Selected feature: Instances with index 0
F1 score for field AvTime: 0.9428076256499134
F1 score for field AvPoints: 0.9428076256499134
F1 score for field StDtime: 0.9428076256499134
F1 score for field StDpoints: 0.9428076256499134
Selected feature: AvTime with index 0
F1 score for field AvPoints: 0.9428076256499134
F1 score for field StDtime: 0.9428076256499134
F1 score for field StDpoints: 0.9428076256499134
Selected feature: AvPoints with index 0
F1 score for field StDtime: 0.9428076256499134
F1 score for field StDpoints: 0.9428076256499134
Selected feature: StDtime with index 0
F1 score for field StDpoints: 0.9428076256499134
Selected feature: StDpoints with index 0
picked_feats_record = ['TotalDwellTime', 'GazePointsAOI', 'Instances', 'AvTime', 'AvPoints', 'StDtime', 'StDpoints']
f1_score_feature_selection = [0.9428076256499134, 0.9428076256499134, 0.9428076256499134, 0.9428076256499134, 0.9428076256499134, 0.9428076256499134, 0.9428076256499134]
\ No newline at end of file
Searching for best clf in {'C': [0.1, 1, 10, 100], 'kernel': ['linear', 'poly', 'rbf', 'sigmoid'], 'gamma': ['auto', 'scale']} with 1521 datapoints with following features: ['TotalDwellTime', 'GazePointsAOI', 'StDtime', 'StDpoints', 'AvTime', 'Instances', 'AvPoints']
Performing Grid Search...
Fitting 5 folds for each of 32 candidates, totalling 160 fits
Seed 7 - Split Variant 0 - Test m-Accuracy: 0.9019607843137255
Performing fit of best model with params: C - 1; Kernel - linear; Gamma - auto on full data set...
Seed 7 - Split Variant 0 - Accuracy: 0.9019607843137255 - F1: 0.9484536082474226 - Recall: 1.0 - MCC: nan
Probability of null hypothesis: 0.001
\ No newline at end of file
saved figs/feature selection bad.png

33 KiB

saved figs/feature selection expert.png

28.6 KiB

saved figs/feature selection good.png

21.4 KiB

saved figs/gaze points.png

552 KiB

This diff is collapsed.
Source diff could not be displayed: it is too large. Options to address this: view the blob.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment