# Electrocardiogram We are dealing with an exteremly imbalance dataset related to electrocardiogram signals that contain binary classes and labeled as good(0) and bad(1) signals. ## STEP 1: Fill missing values All the columns in our data contain missing values a range from 25 to 70. By using `from sklearn.impute import KNNImputer` we fill all of them using 5 of the nearst neighbors of that missing value. ``` imputer = KNNImputer(n_neighbors=5) data_imputed = imputer.fit_transform(data_frame) data_frame_imputed = pandas.DataFrame(data_imputed, columns=columns) missing_value_counts = data_frame_imputed.isna().sum() write_textfile(f"{data_directory}/no_missing.txt", missing_value_counts) return data_frame_imputed ``` ## STEP 2: Scaling We used `from sklearn.preprocessing import RobustScaler` to handle scaling. ``` scaler = RobustScaler() x = data_frame.drop("label", axis=1) x_scale = scaler.fit_transform(x) data_frame_scaled = pandas.DataFrame(x_scale, columns=x.columns) data_frame_scaled["label"] = labels.values ``` ## STEP 3: k-fold cross validation + stratify classes + balancing training data First of all we split the dataset into 2 parts train (85%) and test (15%). For making sure that majority class and imbalanced class distributed fairly we passed `stratify=y` ``` x_train, x_test, y_train, y_test = train_test_split( X, y, test_size=0.15, stratify=y, random_state=42, ) ``` Then, for train dataset we used `from sklearn.model_selection import StratifiedKFold` to this class distribution also apply for train and validation data. ``` skf = StratifiedKFold(n_splits=n_splits, shuffle=True, random_state=random_state) for fold_num, (train_idx, val_idx) in enumerate( tqdm.tqdm(skf.split(X, y), total=skf.n_splits, desc="Training Folds"), start=1 ): X_train, X_val = X.iloc[train_idx], X.iloc[val_idx] y_train, y_val = y.iloc[train_idx], y.iloc[val_idx] ``` and finally we use one of these balancing methods `from imblearn.over_sampling import ADASYN, SMOTE, SVMSMOTE, BorderlineSMOTE, KMeansSMOTE` to augment samples for only train data ``` if smote: if smote_method.lower() == "kmeans": sampler = KMeansSMOTE( k_neighbors=5, cluster_balance_threshold=0.1, random_state=random_state, ) elif smote_method.lower() == "smote": sampler = SMOTE(k_neighbors=5, random_state=random_state) elif smote_method.lower() == "svmsmote": sampler = SVMSMOTE(k_neighbors=5, random_state=random_state) elif smote_method.lower() == "borderline": sampler = BorderlineSMOTE(k_neighbors=5, random_state=random_state) elif smote_method.lower() == "adasyn": sampler = ADASYN(n_neighbors=5, random_state=random_state) else: raise ValueError(f"Unknown smote_method: {smote_method}") X_train, y_train = sampler.fit_resample(X_train, y_train) model.fit(X_train, y_train) ``` ## STEP 4: Train different models to find the best possible approach #### What we are looking for: #### Dangerous: Sick → predicted healthy : high recall score or low FN #### Costly: Healthy → predicted sick : high precision score or low FP ## STEP 5: Current results taken KMEANS_SMOTE: | model | stage | accuracy | f1_macro | f2_macro | recall_macro | precision_macro | f1_class0 | f1_class1 | f2_class0 | f2_class1 | recall_class0 | recall_class1 | precision_class0 | precision_class1 | TP | TN | FP | FN | |-----------------------|-------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|-----|-------|----|----| | CatBoost_balanced_knn10 | train | 0.9843784049402589 | 0.8696686267343388 | 0.8824472728294012 | 0.8916952848998795 | 0.8508242781484853 | 0.9919396338322237 | 0.7473976196364541 | 0.9908276010500254 | 0.7740669446087769 | 0.9900881006639566 | 0.7933024691358025 | 0.9938004847319636 | 0.7078480715650071 | 789 | 26898 | 140 | 19 | | CatBoost_balanced_knn10 | test | 0.9802604802604803 | 0.8348421298822796 | 0.8461546793313885 | 0.8541662696976049 | 0.8176680164072361 | 0.9898162729658793 | 0.6798679867986799 | 0.988757446094471 | 0.703551912568306 | 0.9880528191154894 | 0.7202797202797203 | 0.991586032814472 | 0.64375 | 103 | 4714 | 57 | 40 | | LGBM_KMEANS_SMOTE_knn10 | train | 0.9883286128479746 | 0.8784419356817057 | 0.8436008106620193 | 0.8240767336379762 | 0.9582821430574249 | 0.9940169232360254 | 0.7628669481273861 | 0.9966698960611392 | 0.6905317252628993 | 0.9984466771524954 | 0.6497067901234568 | 0.9896275269971563 | 0.9269367591176938 | 775 | 27036 | 2 | 33 | | LGBM_KMEANS_SMOTE_knn10 | test | 0.9865689865689866 | 0.8543196878009516 | 0.8121616449258658 | 0.7895809912158687 | 0.9600745182511498 | 0.9931221342225928 | 0.7155172413793104 | 0.9964866786565728 | 0.6278366111951589 | 0.9987424020121568 | 0.5804195804195804 | 0.9875647668393782 | 0.9325842696629213 | 83 | 4765 | 6 | 60 | ## Tuning LightGBM and CatBoost As it is written in `models/catboost_model.py` tune function for this model we used the following parameters: ``` scaling_methods = [ "standard_scaling", "robust_scaling", "minmax_scaling", "yeo_johnson", ] sampling_methods = [ "KMeansSMOTE", "class_weight", ] learning_rate_list = [0.03, 0.05, 0.1] depth_list = [6, 8] l2_leaf_reg_list = [1, 3] subsample_list = [0.8, 1.0] k_neighbors_list = [10] kmeans_estimator_list = [5] ``` Also, for `models/lightgbm_model.py` tune function we used the folowing parameters: ``` scaling_methods = [ "standard_scaling", "robust_scaling", "minmax_scaling", "yeo_johnson", ] sampling_methods = [ "KMeansSMOTE", "class_weight", ] boosting_type_list = ["gbdt", "dart"] learning_rate_list = [0.03, 0.05, 0.1] number_of_leaves_list = [100] l2_regularization_lambda_list = [0.1, 0.5] l1_regularization_alpha_list = [0.1, 0.5] tree_subsample_tree_list = [0.8, 1.0] subsample_list = [0.8, 1.0] kmeans_smote_k_neighbors_list = [10] kmeans_smote_n_clusters_list = [5] ``` After tuning we train both models based on their best parameters and compare on an imbalanced test data. here is the comparison results: | model | accuracy | f1_macro | f2_macro | recall_macro | precision_macro | f1_class0 | f2_class0 | recall_class0 | precision_class0 | f1_class1 | f2_class1 | recall_class1 | precision_class1 | TP | TN | FP | FN | |----------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|--------------------|----|------|----|----| | catboost | 0.9814814814814815 | 0.8195693865042805 | 0.8013174756506312 | 0.7903526990720451 | 0.8559205703525894 | 0.9904901243599122 | 0.9921698350221925 | 0.9932928107315029 | 0.9877032096706961 | 0.6486486486486487 | 0.6104651162790697 | 0.5874125874125874 | 0.7241379310344828 | 84 | 4739 | 32 | 59 | | lightgbm | 0.9849409849409849 | 0.8469442386692707 | 0.8185917013944679 | 0.8023094072140393 | 0.9084632979829487 | 0.9922755741127348 | 0.9946427824048885 | 0.9962272060364703 | 0.9883551673944687 | 0.7016129032258065 | 0.6425406203840472 | 0.6083916083916084 | 0.8285714285714286 | 87 | 4753 | 18 | 56 | ## next steps: ``` ✅ 1. Stratified K-fold only apply on train. ✅ 2. train LGBM model using KMEANS_SMOTE with knn k_neighbors=10 (fine-tune remained) ✅ 3. train Cat_boost using KMEANS_SMOTE with knn k_neighbors=10 (fine-tune remained) 🗹 4. implement proposed methods of this article : https://1drv.ms/b/c/ab2a38fe5c318317/IQBEDsSFcYj6R6AMtOnh0X6DAZUlFqAYq19WT8nTeXomFwg 🗹 5. compare proposed model with SMOTE vs oversampling balancing method ```