[2021/04/14-20:02:16.330] [FT-TrainWorkThread-kfX1SEa1-1904] [INFO] [dku.analysis.prediction]  - ******************************************
[2021/04/14-20:02:16.332] [FT-TrainWorkThread-kfX1SEa1-1904] [INFO] [dku.analysis.prediction]  - ** Start train session s8
[2021/04/14-20:02:16.332] [FT-TrainWorkThread-kfX1SEa1-1904] [INFO] [dku.analysis.prediction]  - ******************************************
[2021/04/14-20:02:16.334] [FT-TrainWorkThread-kfX1SEa1-1904] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 5] Search for split: p=type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337 i=bf10335945b1f843a4ae9838d57674b3-0
[2021/04/14-20:02:16.336] [FT-TrainWorkThread-kfX1SEa1-1904] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 7] Checking if splits are up to date. Policy: type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337, instance id: bf10335945b1f843a4ae9838d57674b3-0
[2021/04/14-20:02:16.336] [FT-TrainWorkThread-kfX1SEa1-1904] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 7] Search for split: p=type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337 i=bf10335945b1f843a4ae9838d57674b3-0
[2021/04/14-20:02:16.337] [FT-TrainWorkThread-kfX1SEa1-1904] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 8] Search for split: p=type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337 i=bf10335945b1f843a4ae9838d57674b3-0
[2021/04/14-20:02:16.340] [FT-TrainWorkThread-kfX1SEa1-1904] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 11] Checking if splits are up to date. Policy: type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337, instance id: bf10335945b1f843a4ae9838d57674b3-0
[2021/04/14-20:02:16.340] [FT-TrainWorkThread-kfX1SEa1-1904] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 11] Search for split: p=type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337 i=bf10335945b1f843a4ae9838d57674b3-0
[2021/04/14-20:02:16.342] [FT-TrainWorkThread-kfX1SEa1-1904] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 13] Search for split: p=type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337 i=bf10335945b1f843a4ae9838d57674b3-0
[2021/04/14-20:02:16.345] [FT-TrainWorkThread-kfX1SEa1-1904] [INFO] [dku.analysis.ml.python] T-0IfwLWGQ - Joining processing thread ...
[2021/04/14-20:02:16.347] [MRT-1905] [INFO] [dku.analysis.ml.python]  - TrainAdditionalThread done
[2021/04/14-20:02:16.347] [MRT-1906] [INFO] [dku.analysis.ml.python]  - Running a preprocessing set: pp1 in /data/home/dkuacademy/Library/dss-9.0/design/analysis-data/QS_DATAENG2/oSylVxXB/0IfwLWGQ/sessions/s8/pp1
[2021/04/14-20:02:16.349] [FT-TrainWorkThread-kfX1SEa1-1904] [INFO] [dku.analysis.ml.python] T-0IfwLWGQ - Processing thread joined ...
[2021/04/14-20:02:16.349] [FT-TrainWorkThread-kfX1SEa1-1904] [INFO] [dku.analysis.ml.python] T-0IfwLWGQ - Joining processing thread ...
[2021/04/14-20:02:16.350] [MRT-1906] [INFO] [dku.block.link]  - Started a socket on port 40805
[2021/04/14-20:02:16.353] [MRT-1906] [INFO] [dku.ml.kernel]  - Writing output of python-single-command-kernel to /data/home/dkuacademy/Library/dss-9.0/design/analysis-data/QS_DATAENG2/oSylVxXB/0IfwLWGQ/sessions/s8/pp1/train.log
[2021/04/14-20:02:16.360] [MRT-1906] [INFO] [dku.code.envs.resolution]  - Executing Python activity in builtin env
[2021/04/14-20:02:16.361] [MRT-1906] [WARN] [dku.code.projectLibs]  - External libraries file not found: /data/home/dkuacademy/Library/dss-9.0/design/config/projects/QS_DATAENG2/lib/external-libraries.json
[2021/04/14-20:02:16.361] [MRT-1906] [INFO] [dku.code.projectLibs]  - EXTERNAL LIBS FROM QS_DATAENG2 is {"gitReferences":{},"pythonPath":["python"],"rsrcPath":["R"],"importLibrariesFromProjects":[]}
[2021/04/14-20:02:16.361] [MRT-1906] [INFO] [dku.code.projectLibs]  - chunkFolder is /data/home/dkuacademy/Library/dss-9.0/design/config/projects/QS_DATAENG2/lib/R
[2021/04/14-20:02:16.361] [MRT-1906] [INFO] [dku.python.single_command.kernel]  - Starting Python process for kernel  python-single-command-kernel
[2021/04/14-20:02:16.361] [MRT-1906] [INFO] [dip.tickets]  - Creating API ticket for analysis-ml-QS_DATAENG2-aU1av0J on behalf of tara id=analysis-ml-QS_DATAENG2-aU1av0J_TIoMVaxEshcA
[2021/04/14-20:02:16.362] [MRT-1906] [INFO] [dku.security.process]  - Starting process (regular)
[2021/04/14-20:02:16.364] [MRT-1906] [INFO] [dku.security.process]  - Process started with pid=34616
[2021/04/14-20:02:16.364] [MRT-1906] [INFO] [dku.processes.cgroups]  - Will use cgroups []
[2021/04/14-20:02:16.365] [MRT-1906] [INFO] [dku.processes.cgroups]  - Applying rules to used cgroups: []
[2021/04/14-20:02:16.367] [KNL-python-single-command-kernel-monitor-1909] [INFO] [dku.resourceusage]  - Reporting start of CRU:{"context":{"type":"ANALYSIS_ML_TRAIN","authIdentifier":"tara","projectKey":"QS_DATAENG2","analysisId":"oSylVxXB","mlTaskId":"0IfwLWGQ","sessionId":"s8"},"type":"LOCAL_PROCESS","id":"fpYbjO7yDnRl2JsM","startTime":1618423336367,"localProcess":{"cpuCurrent":0.0}}
[2021/04/14-20:02:16.369] [process-resource-monitor-34616-1910] [DEBUG] [dku.resource]  - Process stats for pid 34616: {"pid":34616,"commandName":"/data/home/dkuacademy/Library/dss-9.0/design/bin/python","cpuUserTimeMS":0,"cpuSystemTimeMS":0,"cpuChildrenUserTimeMS":0,"cpuChildrenSystemTimeMS":0,"cpuTotalMS":0,"cpuCurrent":0.0,"vmSizeMB":120,"vmRSSMB":3,"vmHWMMB":3,"vmRSSAnonMB":1,"vmDataMB":1,"vmSizePeakMB":120,"vmRSSPeakMB":3,"vmRSSTotalMBS":0,"majorFaults":0,"childrenMajorFaults":0}
Installing debugging signal handler
[2021-04-14 20:02:17,224] [34616/MainThread] [INFO] [dataiku.base.socket_block_link] Connecting to localhost (127.0.0.1) at port 40805
[2021-04-14 20:02:17,224] [34616/MainThread] [INFO] [dataiku.base.socket_block_link] Connected to localhost (127.0.0.1) at port 40805
[2021/04/14-20:02:17.226] [MRT-1906] [INFO] [dku.link.secret_protected]  - Connected to kernel
[2021/04/14-20:02:17.230] [MRT-1906] [INFO] [dku.block.link.interaction]  - Execute link command respClazz=true respTypeToken=false respIsString=false is=false asyncInputStream=false os=false
[2021-04-14 20:02:19,280] [34616/MainThread] [INFO] [dataiku.doctor.utils.dku_pickle] Setting cloudpickle as the pickling tool
/data/home/dkuacademy/Applications/dataiku-dss-9.0.0/python/dataiku/doctor/dkuapi.py:16: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()
  argspec = inspect.getargspec(api)
[2021-04-14 20:02:19,340] [34616/MainThread] [INFO] [root] Running analysis command: train_prediction_models_nosave
[2021-04-14 20:02:19,341] [34616/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: DatasetSanityCheckDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_DATASET_SANITY_CHECKS
[2021-04-14 20:02:19,341] [34616/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: ClassifierAccuracyCheckDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_MODEL_CHECK
[2021-04-14 20:02:19,341] [34616/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: RegressionR2CheckDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_MODEL_CHECK
[2021-04-14 20:02:19,341] [34616/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: LeakageDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_LEAKAGE_DETECTION
[2021-04-14 20:02:19,341] [34616/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: TreeOverfitDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_TRAINING_OVERFIT
[2021-04-14 20:02:19,341] [34616/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: MLAssertionsDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_ML_ASSERTIONS
[2021-04-14 20:02:19,341] [34616/MainThread] [INFO] [dataiku.doctor.commands] PPS is {'target_remapping': [], 'skipPreprocessing': False, 'per_feature': {'FlightNum': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Origin': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 100, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'NONE', 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Text'}, 'autoReason': 'REJECT_CARDINALITY', 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'dep_hour': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Late': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 5, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'DROP', 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Boolean'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Dest': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 100, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'NONE', 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Text'}, 'autoReason': 'REJECT_CARDINALITY', 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'dep_dow': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'AirTime': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'UniqueCarrier': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 5, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'DROP', 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Text'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Late_avg': {'generate_derivative': False, 'impute_constant_value': 0.0, 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'TARGET', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'DoubleMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'dep_month': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'dep_woy': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Distance': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}}, 'reduce': {'enabled': False, 'kept_variance': 0.0}, 'feature_generation': {'pairwise_linear': {'behavior': 'DISABLED'}, 'polynomial_combinations': {'behavior': 'DISABLED'}, 'manual_interactions': {'interactions': []}, 'numericals_clustering': {'k': 0, 'all_features': False, 'input_features': [], 'behavior': 'DISABLED'}, 'categoricals_count_transformer': {'all_features': False, 'input_features': [], 'behavior': 'DISABLED'}}, 'feature_selection_params': {'method': 'NONE', 'random_forest_params': {'n_trees': 30, 'depth': 10, 'n_features': 25}, 'lasso_params': {'alpha': [0.01, 0.1, 1.0, 10.0, 100.0], 'cross_validate': True}, 'pca_params': {'n_features': 25, 'variance_proportion': 0.9}, 'correlation_params': {'min_abs_correlation': 0.0, 'max_abs_correlation': 1.0, 'n_features': 25}, 'custom_params': {'code': '# type your code here'}}, 'preprocessingFitSampleRatio': 1.0, 'preprocessingFitSampleSeed': 1337}
[2021-04-14 20:02:19,341] [34616/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Loading train set
[2021-04-14 20:02:19,379] [34616/MainThread] [INFO] [root] Reading with dtypes: None
[2021-04-14 20:02:19,379] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_month: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 20:02:19,379] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_woy: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 20:02:19,379] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_dow: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 20:02:19,379] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_hour: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 20:02:19,379] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for UniqueCarrier: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 20:02:19,379] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for FlightNum: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 20:02:19,379] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for AirTime: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 20:02:19,379] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Late: str (schema_type=boolean feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 20:02:19,379] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Origin: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 20:02:19,379] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Dest: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 20:02:19,379] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Distance: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 20:02:19,379] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Late_avg: <class 'numpy.float64'> (schema_type=double feature_type=NUMERIC feature_role=TARGET)
[2021-04-14 20:02:19,379] [34616/MainThread] [INFO] [root] Reading with FIXED dtypes: {'dep_month': <class 'numpy.float64'>, 'dep_woy': <class 'numpy.float64'>, 'dep_dow': <class 'numpy.float64'>, 'dep_hour': <class 'numpy.float64'>, 'UniqueCarrier': 'str', 'FlightNum': <class 'numpy.float64'>, 'AirTime': <class 'numpy.float64'>, 'Late': 'str', 'Origin': 'str', 'Dest': 'str', 'Distance': <class 'numpy.float64'>, 'Late_avg': <class 'numpy.float64'>}
[2021-04-14 20:02:19,491] [34616/MainThread] [INFO] [root] Loaded table
[2021-04-14 20:02:19,492] [34616/MainThread] [INFO] [dataiku.doctor.utils]  Coercion done
[2021-04-14 20:02:19,492] [34616/MainThread] [INFO] [dataiku.doctor.utils.split] Loaded train df: shape=(79912,12)
[2021-04-14 20:02:19,492] [34616/MainThread] [INFO] [dataiku.doctor.commands] Train col : dep_month (float64)
[2021-04-14 20:02:19,492] [34616/MainThread] [INFO] [dataiku.doctor.commands] Train col : dep_woy (float64)
[2021-04-14 20:02:19,492] [34616/MainThread] [INFO] [dataiku.doctor.commands] Train col : dep_dow (float64)
[2021-04-14 20:02:19,492] [34616/MainThread] [INFO] [dataiku.doctor.commands] Train col : dep_hour (float64)
[2021-04-14 20:02:19,492] [34616/MainThread] [INFO] [dataiku.doctor.commands] Train col : UniqueCarrier (object)
[2021-04-14 20:02:19,492] [34616/MainThread] [INFO] [dataiku.doctor.commands] Train col : FlightNum (float64)
[2021-04-14 20:02:19,492] [34616/MainThread] [INFO] [dataiku.doctor.commands] Train col : AirTime (float64)
[2021-04-14 20:02:19,492] [34616/MainThread] [INFO] [dataiku.doctor.commands] Train col : Late (object)
[2021-04-14 20:02:19,492] [34616/MainThread] [INFO] [dataiku.doctor.commands] Train col : Origin (object)
[2021-04-14 20:02:19,492] [34616/MainThread] [INFO] [dataiku.doctor.commands] Train col : Dest (object)
[2021-04-14 20:02:19,492] [34616/MainThread] [INFO] [dataiku.doctor.commands] Train col : Distance (float64)
[2021-04-14 20:02:19,492] [34616/MainThread] [INFO] [dataiku.doctor.commands] Train col : Late_avg (float64)
[2021-04-14 20:02:19,493] [34616/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Loading train set
[2021-04-14 20:02:19,493] [34616/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Loading test set
[2021-04-14 20:02:19,493] [34616/MainThread] [INFO] [root] Reading with dtypes: None
[2021-04-14 20:02:19,493] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_month: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 20:02:19,493] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_woy: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 20:02:19,493] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_dow: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 20:02:19,493] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_hour: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 20:02:19,493] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for UniqueCarrier: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 20:02:19,493] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for FlightNum: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 20:02:19,493] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for AirTime: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 20:02:19,493] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Late: str (schema_type=boolean feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 20:02:19,493] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Origin: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 20:02:19,493] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Dest: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 20:02:19,493] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Distance: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 20:02:19,493] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Late_avg: <class 'numpy.float64'> (schema_type=double feature_type=NUMERIC feature_role=TARGET)
[2021-04-14 20:02:19,493] [34616/MainThread] [INFO] [root] Reading with FIXED dtypes: {'dep_month': <class 'numpy.float64'>, 'dep_woy': <class 'numpy.float64'>, 'dep_dow': <class 'numpy.float64'>, 'dep_hour': <class 'numpy.float64'>, 'UniqueCarrier': 'str', 'FlightNum': <class 'numpy.float64'>, 'AirTime': <class 'numpy.float64'>, 'Late': 'str', 'Origin': 'str', 'Dest': 'str', 'Distance': <class 'numpy.float64'>, 'Late_avg': <class 'numpy.float64'>}
[2021-04-14 20:02:19,523] [34616/MainThread] [INFO] [root] Loaded table
[2021-04-14 20:02:19,524] [34616/MainThread] [INFO] [dataiku.doctor.utils]  Coercion done
[2021-04-14 20:02:19,524] [34616/MainThread] [INFO] [dataiku.doctor.utils.split] Loaded test df: shape=(20088,12)
[2021-04-14 20:02:19,524] [34616/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Loading test set
[2021-04-14 20:02:19,524] [34616/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Collecting statistics
[2021-04-14 20:02:19,524] [34616/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at FlightNum... (type=NUMERIC)
[2021-04-14 20:02:19,525] [34616/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-14 20:02:19,528] [34616/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Origin... (type=CATEGORY)
[2021-04-14 20:02:19,541] [34616/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at dep_hour... (type=NUMERIC)
[2021-04-14 20:02:19,541] [34616/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-14 20:02:19,544] [34616/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Late... (type=CATEGORY)
[2021-04-14 20:02:19,555] [34616/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Dest... (type=CATEGORY)
[2021-04-14 20:02:19,567] [34616/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at dep_dow... (type=NUMERIC)
[2021-04-14 20:02:19,567] [34616/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-14 20:02:19,569] [34616/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at AirTime... (type=NUMERIC)
[2021-04-14 20:02:19,569] [34616/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-14 20:02:19,572] [34616/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at UniqueCarrier... (type=CATEGORY)
[2021-04-14 20:02:19,583] [34616/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Late_avg... (type=NUMERIC)
[2021-04-14 20:02:19,583] [34616/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at dep_month... (type=NUMERIC)
[2021-04-14 20:02:19,583] [34616/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-14 20:02:19,586] [34616/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at dep_woy... (type=NUMERIC)
[2021-04-14 20:02:19,586] [34616/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-14 20:02:19,588] [34616/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Distance... (type=NUMERIC)
[2021-04-14 20:02:19,588] [34616/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-14 20:02:19,591] [34616/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Collecting statistics
[2021-04-14 20:02:19,591] [34616/MainThread] [INFO] [dataiku.doctor.multiframe] generating interactions
[2021-04-14 20:02:19,591] [34616/MainThread] [INFO] [dataiku.doctor.multiframe] {'target_remapping': [], 'skipPreprocessing': False, 'per_feature': {'FlightNum': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Origin': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 100, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'NONE', 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Text'}, 'autoReason': 'REJECT_CARDINALITY', 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'dep_hour': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Late': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 5, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'DROP', 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Boolean'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Dest': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 100, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'NONE', 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Text'}, 'autoReason': 'REJECT_CARDINALITY', 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'dep_dow': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'AirTime': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'UniqueCarrier': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 5, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'DROP', 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Text'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Late_avg': {'generate_derivative': False, 'impute_constant_value': 0.0, 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'TARGET', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'DoubleMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'dep_month': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'dep_woy': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Distance': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}}, 'reduce': {'enabled': False, 'kept_variance': 0.0}, 'feature_generation': {'pairwise_linear': {'behavior': 'DISABLED'}, 'polynomial_combinations': {'behavior': 'DISABLED'}, 'manual_interactions': {'interactions': []}, 'numericals_clustering': {'k': 0, 'all_features': False, 'input_features': [], 'behavior': 'DISABLED'}, 'categoricals_count_transformer': {'all_features': False, 'input_features': [], 'behavior': 'DISABLED'}}, 'feature_selection_params': {'method': 'NONE', 'random_forest_params': {'n_trees': 30, 'depth': 10, 'n_features': 25}, 'lasso_params': {'alpha': [0.01, 0.1, 1.0, 10.0, 100.0], 'cross_validate': True}, 'pca_params': {'n_features': 25, 'variance_proportion': 0.9}, 'correlation_params': {'min_abs_correlation': 0.0, 'max_abs_correlation': 1.0, 'n_features': 25}, 'custom_params': {'code': '# type your code here'}}, 'preprocessingFitSampleRatio': 1.0, 'preprocessingFitSampleSeed': 1337}
[2021-04-14 20:02:19,591] [34616/MainThread] [INFO] [dataiku.doctor.multiframe] No feature selection to perform
[2021-04-14 20:02:19,591] [34616/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Preprocessing train set
[2021-04-14 20:02:19,592] [34616/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 79912
[2021-04-14 20:02:19,592] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RemapValueToOutput
[2021-04-14 20:02:19,592] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:MultipleImputeMissingFromInput
[2021-04-14 20:02:19,592] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {'FlightNum': 2149.660088597457, 'dep_hour': 13.086732906196817, 'dep_dow': 3.9422364601061166, 'AirTime': 103.37577585343878, 'dep_month': 3.7664931424567025, 'dep_woy': 14.544386324957452, 'Distance': 724.2432175392933}
[2021-04-14 20:02:19,597] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (FlightNum)
[2021-04-14 20:02:19,597] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale FlightNum (avg=2149.660088597457 std=1944.0927463213502 shift=2149.660088597457 inv_scale=0.0005143787516784985)
[2021-04-14 20:02:19,629] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled FlightNum (avg=3.871408205081647e-16 std=1.0000000000000007) nulls=0
[2021-04-14 20:02:19,629] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (dep_hour)
[2021-04-14 20:02:19,630] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_hour (avg=13.086732906196817 std=4.8018824732181775 shift=13.086732906196817 inv_scale=0.20825166079706428)
[2021-04-14 20:02:19,632] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_hour (avg=6.6931254923342605e-16 std=0.9999999999996343) nulls=0
[2021-04-14 20:02:19,632] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (dep_dow)
[2021-04-14 20:02:19,632] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_dow (avg=3.9422364601061166 std=1.9826725071033415 shift=3.9422364601061166 inv_scale=0.5043697314696651)
[2021-04-14 20:02:19,634] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_dow (avg=1.696354984046843e-15 std=1.000000000000285) nulls=0
[2021-04-14 20:02:19,634] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (AirTime)
[2021-04-14 20:02:19,635] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale AirTime (avg=103.37577585343878 std=68.48571355586151 shift=103.37577585343878 inv_scale=0.014601585470586261)
[2021-04-14 20:02:19,636] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled AirTime (avg=5.50103338372283e-15 std=1.0000000000000062) nulls=0
[2021-04-14 20:02:19,637] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (dep_month)
[2021-04-14 20:02:19,637] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_month (avg=3.7664931424567025 std=1.591910424424286 shift=3.7664931424567025 inv_scale=0.6281760485120572)
[2021-04-14 20:02:19,639] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_month (avg=1.1532441946155217e-13 std=1.0000000000012612) nulls=0
[2021-04-14 20:02:19,639] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (dep_woy)
[2021-04-14 20:02:19,639] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_woy (avg=14.544386324957452 std=6.971213818152402 shift=14.544386324957452 inv_scale=0.14344704180441167)
[2021-04-14 20:02:19,642] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_woy (avg=2.0155730012125603e-13 std=1.0000000000000655) nulls=0
[2021-04-14 20:02:19,643] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (Distance)
[2021-04-14 20:02:19,643] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale Distance (avg=724.2432175392933 std=561.8104116867048 shift=724.2432175392933 inv_scale=0.0017799598925155786)
[2021-04-14 20:02:19,646] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled Distance (avg=1.1686294916829722e-16 std=0.9999999999999956) nulls=0
[2021-04-14 20:02:19,646] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FlushDFBuilder(num_flagonly)
[2021-04-14 20:02:19,646] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FastSparseDummifyProcessor (Origin)
[2021-04-14 20:02:19,674] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(79912, 103) nnz=79912
[2021-04-14 20:02:19,675] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FastSparseDummifyProcessor (Late)
[2021-04-14 20:02:19,690] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(79912, 2) nnz=60354
[2021-04-14 20:02:19,691] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FastSparseDummifyProcessor (Dest)
[2021-04-14 20:02:19,709] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(79912, 103) nnz=79912
[2021-04-14 20:02:19,710] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FastSparseDummifyProcessor (UniqueCarrier)
[2021-04-14 20:02:19,724] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(79912, 7) nnz=41098
[2021-04-14 20:02:19,725] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:MultipleImputeMissingFromInput
[2021-04-14 20:02:19,725] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2021-04-14 20:02:19,725] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FlushDFBuilder(cat_flagpresence)
[2021-04-14 20:02:19,725] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:MultipleImputeMissingFromInput
[2021-04-14 20:02:19,726] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2021-04-14 20:02:19,726] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FlushDFBuilder(interaction)
[2021-04-14 20:02:19,726] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RealignTarget
[2021-04-14 20:02:19,726] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] Realign target series = (79912,)
[2021-04-14 20:02:19,729] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] After realign target: (79912,)
[2021-04-14 20:02:19,729] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:DropRowsWhereNoTarget
[2021-04-14 20:02:19,730] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] Deleting 0 rows because one of ['target'] is missing
[2021-04-14 20:02:19,730] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] MF before = (79912, 222)
[2021-04-14 20:02:19,730] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] target before = (79912,)
[2021-04-14 20:02:19,734] [34616/MainThread] [INFO] [dataiku.doctor.multiframe] MultiFrame, dropping rows: []
[2021-04-14 20:02:19,764] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] After DRWNT input_df=(79912, 12)
[2021-04-14 20:02:19,764] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] MF after = (79912, 222)
[2021-04-14 20:02:19,764] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] target after = (79912,)
[2021-04-14 20:02:19,764] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:DumpPipelineState
[2021-04-14 20:02:19,764] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] ********* Pipeline state (Before feature selection)
[2021-04-14 20:02:19,764] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]    input_df= (79912, 12) 
[2021-04-14 20:02:19,764] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]    current_mf=(79912, 222) 
[2021-04-14 20:02:19,764] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]    PPR: 
[2021-04-14 20:02:19,764] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]       target = <class 'pandas.core.series.Series'> ((79912,))
[2021-04-14 20:02:19,764] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:EmitCurrentMFAsResult
[2021-04-14 20:02:19,764] [34616/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 79912
[2021-04-14 20:02:19,764] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:DumpPipelineState
[2021-04-14 20:02:19,764] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] ********* Pipeline state (At end)
[2021-04-14 20:02:19,765] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]    input_df= (79912, 12) 
[2021-04-14 20:02:19,765] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]    current_mf=(0, 0) 
[2021-04-14 20:02:19,765] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]    PPR: 
[2021-04-14 20:02:19,765] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]       target = <class 'pandas.core.series.Series'> ((79912,))
[2021-04-14 20:02:19,765] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]       TRAIN = <class 'dataiku.doctor.multiframe.MultiFrame'> ((79912, 222))
[2021-04-14 20:02:19,765] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]       UNPROCESSED = <class 'pandas.core.frame.DataFrame'> ((79912, 12))
[2021-04-14 20:02:19,766] [34616/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Preprocessing train set
[2021-04-14 20:02:19,766] [34616/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Preprocessing test set
[2021-04-14 20:02:19,766] [34616/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 20088
[2021-04-14 20:02:19,766] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RemapValueToOutput
[2021-04-14 20:02:19,766] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:MultipleImputeMissingFromInput
[2021-04-14 20:02:19,766] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {'FlightNum': 2149.660088597457, 'dep_hour': 13.086732906196817, 'dep_dow': 3.9422364601061166, 'AirTime': 103.37577585343878, 'dep_month': 3.7664931424567025, 'dep_woy': 14.544386324957452, 'Distance': 724.2432175392933}
[2021-04-14 20:02:19,768] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (FlightNum)
[2021-04-14 20:02:19,768] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale FlightNum (avg=2129.2614496216647 std=1938.800057332436 shift=2149.660088597457 inv_scale=0.0005143787516784985)
[2021-04-14 20:02:19,769] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled FlightNum (avg=-0.01049262645230846 std=0.9972775532448616) nulls=0
[2021-04-14 20:02:19,769] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (dep_hour)
[2021-04-14 20:02:19,770] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_hour (avg=13.137544802867383 std=4.811521844053324 shift=13.086732906196817 inv_scale=0.20825166079706428)
[2021-04-14 20:02:19,771] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_hour (avg=0.010581661869894975 std=1.0020074149854938) nulls=0
[2021-04-14 20:02:19,771] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (dep_dow)
[2021-04-14 20:02:19,771] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_dow (avg=3.9275189167662288 std=1.9835343957941987 shift=3.9422364601061166 inv_scale=0.5043697314696651)
[2021-04-14 20:02:19,772] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_dow (avg=-0.007423083382236679 std=1.0004347105675437) nulls=0
[2021-04-14 20:02:19,772] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (AirTime)
[2021-04-14 20:02:19,772] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale AirTime (avg=103.51359020310633 std=68.67809152402859 shift=103.37577585343878 inv_scale=0.014601585470586261)
[2021-04-14 20:02:19,773] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled AirTime (avg=0.002012308005743753 std=1.0028090233448395) nulls=0
[2021-04-14 20:02:19,773] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (dep_month)
[2021-04-14 20:02:19,774] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_month (avg=3.759856630824373 std=1.5818950705064383 shift=3.7664931424567025 inv_scale=0.6281760485120572)
[2021-04-14 20:02:19,781] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_month (avg=-0.004168897653024282 std=0.9937085945514825) nulls=0
[2021-04-14 20:02:19,781] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (dep_woy)
[2021-04-14 20:02:19,781] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_woy (avg=14.508910792512943 std=6.937862611767304 shift=14.544386324957452 inv_scale=0.14344704180441167)
[2021-04-14 20:02:19,782] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_woy (avg=-0.005088860185616315 std=0.9952158681034459) nulls=0
[2021-04-14 20:02:19,782] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (Distance)
[2021-04-14 20:02:19,783] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale Distance (avg=726.2480585424133 std=562.1976507268723 shift=724.2432175392933 inv_scale=0.0017799598925155786)
[2021-04-14 20:02:19,784] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled Distance (avg=0.0035685365764242215 std=1.0006892699603191) nulls=0
[2021-04-14 20:02:19,784] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FlushDFBuilder(num_flagonly)
[2021-04-14 20:02:19,784] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FastSparseDummifyProcessor (Origin)
[2021-04-14 20:02:19,790] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(20088, 103) nnz=20088
[2021-04-14 20:02:19,790] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FastSparseDummifyProcessor (Late)
[2021-04-14 20:02:19,795] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(20088, 2) nnz=15203
[2021-04-14 20:02:19,796] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FastSparseDummifyProcessor (Dest)
[2021-04-14 20:02:19,801] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(20088, 103) nnz=20088
[2021-04-14 20:02:19,801] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FastSparseDummifyProcessor (UniqueCarrier)
[2021-04-14 20:02:19,805] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(20088, 7) nnz=10425
[2021-04-14 20:02:19,806] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:MultipleImputeMissingFromInput
[2021-04-14 20:02:19,806] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2021-04-14 20:02:19,806] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FlushDFBuilder(cat_flagpresence)
[2021-04-14 20:02:19,806] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:MultipleImputeMissingFromInput
[2021-04-14 20:02:19,806] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2021-04-14 20:02:19,806] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FlushDFBuilder(interaction)
[2021-04-14 20:02:19,806] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RealignTarget
[2021-04-14 20:02:19,806] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] Realign target series = (20088,)
[2021-04-14 20:02:19,807] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] After realign target: (20088,)
[2021-04-14 20:02:19,807] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:DropRowsWhereNoTarget
[2021-04-14 20:02:19,807] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] Deleting 0 rows because one of ['target'] is missing
[2021-04-14 20:02:19,807] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] MF before = (20088, 222)
[2021-04-14 20:02:19,807] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] target before = (20088,)
[2021-04-14 20:02:19,808] [34616/MainThread] [INFO] [dataiku.doctor.multiframe] MultiFrame, dropping rows: []
[2021-04-14 20:02:19,815] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] After DRWNT input_df=(20088, 12)
[2021-04-14 20:02:19,815] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] MF after = (20088, 222)
[2021-04-14 20:02:19,816] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] target after = (20088,)
[2021-04-14 20:02:19,816] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:DumpPipelineState
[2021-04-14 20:02:19,816] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] ********* Pipeline state (Before feature selection)
[2021-04-14 20:02:19,816] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]    input_df= (20088, 12) 
[2021-04-14 20:02:19,816] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]    current_mf=(20088, 222) 
[2021-04-14 20:02:19,816] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]    PPR: 
[2021-04-14 20:02:19,816] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]       target = <class 'pandas.core.series.Series'> ((20088,))
[2021-04-14 20:02:19,816] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:EmitCurrentMFAsResult
[2021-04-14 20:02:19,816] [34616/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 20088
[2021-04-14 20:02:19,816] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:DumpPipelineState
[2021-04-14 20:02:19,816] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] ********* Pipeline state (At end)
[2021-04-14 20:02:19,816] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]    input_df= (20088, 12) 
[2021-04-14 20:02:19,816] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]    current_mf=(0, 0) 
[2021-04-14 20:02:19,816] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]    PPR: 
[2021-04-14 20:02:19,816] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]       target = <class 'pandas.core.series.Series'> ((20088,))
[2021-04-14 20:02:19,816] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]       TRAIN = <class 'dataiku.doctor.multiframe.MultiFrame'> ((20088, 222))
[2021-04-14 20:02:19,816] [34616/MainThread] [DEBUG] [dku.ml.preprocessing]       UNPROCESSED = <class 'pandas.core.frame.DataFrame'> ((20088, 12))
[2021-04-14 20:02:19,816] [34616/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Preprocessing test set
[2021-04-14 20:02:19,817] [34616/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Hyperparameter searching
[2021-04-14 20:02:19,826] [34616/MainThread] [DEBUG] [dku.ml.preprocessing] Deleting 0 rows
[2021-04-14 20:02:19,826] [34616/MainThread] [INFO] [dataiku.doctor.multiframe] MultiFrame, dropping rows: []
[2021-04-14 20:02:19,840] [34616/MainThread] [INFO] [dataiku.doctor.prediction.common] prepare multiframe shape=(79912,222) tn=17740464 nnz=820660 fill_ratio=0.05
[2021-04-14 20:02:19,840] [34616/MainThread] [INFO] [dataiku.doctor.prediction.common] too small, using array
[2021-04-14 20:02:19,951] [34616/MainThread] [INFO] [dataiku.doctor.prediction.common] Create CLF from params: {'algorithm': 'RIDGE_REGRESSION', 'ridge_grid': {'alphaMode': 'MANUAL', 'alpha': {'gridMode': 'EXPLICIT', 'randomMode': 'RANGE', 'limit': {'min': 1e-23}, 'values': [0.1, 1.0, 3.0], 'range': {'min': 0.1, 'max': 3.0, 'nbValues': 3, 'scaling': 'LOGARITHMIC'}}, 'enabled': True}, 'max_ensemble_nodes_serialized': 0, 'metrics': {'evaluationMetric': 'R2', 'customEvaluationMetricGIB': True, 'customEvaluationMetricNeedsProba': False, 'costMatrixWeights': {'tpGain': 1.0, 'tnGain': 0.0, 'fpGain': -0.3, 'fnGain': 0.0}, 'liftPoint': 0.4}, 'autoOptimizeThreshold': True, 'forcedClassifierThreshold': 0.0, 'gridLength': 3, 'grid_search_params': {'mode': 'KFOLD', 'splitRatio': 0.8, 'shuffleIterations': 1, 'nFolds': 5, 'stratified': True, 'strategy': 'GRID', 'bayesianOptimizer': 'SCIKIT_OPTIMIZE', 'randomized': True, 'seed': 0, 'nIter': 0, 'nIterRandom': 3, 'timeout': 0, 'nJobs': 4, 'distributed': False, 'nContainers': 4}, 'pluginAlgoCustomGridSearch': False, 'computeLearningCurves': False, 'skipExpensiveReports': False} for algorithm RIDGE_REGRESSION
[2021-04-14 20:02:19,951] [34616/MainThread] [INFO] [dataiku.doctor.prediction.common] Using K-Fold CV with k=5
[2021-04-14 20:02:19,963] [34616/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluation_monitor] No distributed container configuration is available to run this search
[2021-04-14 20:02:19,968] [34616/MainThread] [INFO] [dataiku.doctor.crossval.search_runner] Fitting 5 folds for each of 3 candidates, totalling 15 fits
[2021-04-14 20:02:19,968] [34616/MainThread] [INFO] [dataiku.doctor.crossval.search_runner] Execute hyperparameter search locally on 4 threads
[2021-04-14 20:02:19,969] [34616/MainThread] [INFO] [dataiku.doctor.crossval.strategies.grid_search_strategy] Running GridSearchStrategy for hyperparameters space: <dataiku.doctor.prediction.common.HyperparametersSpace object at 0x7fb3eab98390>
[2021-04-14 20:02:19,969] [34616/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 0: {'alpha': 0.1}
[2021-04-14 20:02:19,969] [34616/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.local_worker] Starting worker: local-0
[2021-04-14 20:02:19,969] [34616/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 1: {'alpha': 0.1}
[2021-04-14 20:02:19,969] [34616/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.local_worker] Started worker: local-0
[2021-04-14 20:02:19,969] [34616/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.local_worker] Starting worker: local-1
[2021-04-14 20:02:19,969] [34616/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 20:02:19,969] [34616/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 2: {'alpha': 0.1}
[2021-04-14 20:02:19,969] [34616/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.local_worker] Started worker: local-1
[2021-04-14 20:02:19,970] [34616/Thread-2:local-0] [INFO] [dataiku.doctor.crossval.search_context] Fit s=0: alpha=0.1 .......................................................
[2021-04-14 20:02:19,970] [34616/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.local_worker] Starting worker: local-2
[2021-04-14 20:02:19,970] [34616/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 20:02:19,970] [34616/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 3: {'alpha': 0.1}
[2021-04-14 20:02:19,970] [34616/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.local_worker] Started worker: local-2
[2021-04-14 20:02:19,970] [34616/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Fit s=1: alpha=0.1 .......................................................
[2021-04-14 20:02:19,971] [34616/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.local_worker] Starting worker: local-3
[2021-04-14 20:02:19,971] [34616/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 4: {'alpha': 0.1}
[2021-04-14 20:02:19,971] [34616/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 20:02:19,971] [34616/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.local_worker] Started worker: local-3
[2021-04-14 20:02:19,971] [34616/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Fit s=2: alpha=0.1 .......................................................
[2021-04-14 20:02:19,971] [34616/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 20:02:19,972] [34616/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Fit s=3: alpha=0.1 .......................................................
[2021-04-14 20:02:20,666] [34616/Thread-2:local-0] [INFO] [dataiku.doctor.crossval.search_context] Done s=0: alpha=0.1 (ft=0.6s st=0.1s sc=0.8457393757743862, sg=1)
[2021-04-14 20:02:20,685] [34616/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 20:02:20,686] [34616/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 20:02:20,686] [34616/Thread-2:local-0] [INFO] [dataiku.doctor.crossval.search_context] Fit s=4: alpha=0.1 .......................................................
[2021-04-14 20:02:20,689] [34616/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Done s=1: alpha=0.1 (ft=0.6s st=0.1s sc=0.8538902972183009, sg=1)
[2021-04-14 20:02:20,706] [34616/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 0: {'alpha': 3.0}
[2021-04-14 20:02:20,707] [34616/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Done s=3: alpha=0.1 (ft=0.6s st=0.1s sc=0.8449398620182682, sg=1)
[2021-04-14 20:02:20,723] [34616/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 20:02:20,724] [34616/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Done s=2: alpha=0.1 (ft=0.7s st=0.1s sc=0.8490534413428655, sg=1)
[2021-04-14 20:02:20,744] [34616/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 20:02:20,744] [34616/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 1: {'alpha': 3.0}
[2021-04-14 20:02:20,744] [34616/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 20:02:20,745] [34616/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Fit s=0: alpha=3.0 .......................................................
[2021-04-14 20:02:20,745] [34616/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 20:02:20,745] [34616/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Fit s=1: alpha=3.0 .......................................................
[2021-04-14 20:02:20,745] [34616/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 2: {'alpha': 3.0}
[2021-04-14 20:02:20,744] [34616/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 20:02:20,761] [34616/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 20:02:20,761] [34616/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 3: {'alpha': 3.0}
[2021-04-14 20:02:20,762] [34616/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Fit s=2: alpha=3.0 .......................................................
[2021-04-14 20:02:21,593] [34616/Thread-2:local-0] [INFO] [dataiku.doctor.crossval.search_context] Done s=4: alpha=0.1 (ft=0.9s st=0.0s sc=0.8457860195038889, sg=1)
[2021-04-14 20:02:21,606] [34616/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 20:02:21,607] [34616/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 20:02:21,607] [34616/Thread-2:local-0] [INFO] [dataiku.doctor.crossval.search_context] Fit s=3: alpha=3.0 .......................................................
[2021-04-14 20:02:21,640] [34616/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 4: {'alpha': 3.0}
[2021-04-14 20:02:21,853] [34616/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Done s=0: alpha=3.0 (ft=1.1s st=0.0s sc=0.42888276388958213, sg=1)
[2021-04-14 20:02:21,873] [34616/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 20:02:21,873] [34616/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 20:02:21,874] [34616/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Fit s=4: alpha=3.0 .......................................................
[2021-04-14 20:02:21,899] [34616/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 0: {'alpha': 1.0}
[2021-04-14 20:02:21,900] [34616/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Done s=1: alpha=3.0 (ft=1.0s st=0.2s sc=0.43234126047484256, sg=1)
[2021-04-14 20:02:21,917] [34616/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Done s=2: alpha=3.0 (ft=1.0s st=0.1s sc=0.4324344050447977, sg=1)
[2021-04-14 20:02:21,932] [34616/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 20:02:21,932] [34616/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 1: {'alpha': 1.0}
[2021-04-14 20:02:21,932] [34616/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 20:02:21,932] [34616/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Fit s=0: alpha=1.0 .......................................................
[2021-04-14 20:02:21,945] [34616/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 20:02:21,948] [34616/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 2: {'alpha': 1.0}
[2021-04-14 20:02:21,952] [34616/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 20:02:21,952] [34616/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Fit s=1: alpha=1.0 .......................................................
[2021-04-14 20:02:22,186] [34616/Thread-2:local-0] [INFO] [dataiku.doctor.crossval.search_context] Done s=3: alpha=3.0 (ft=0.5s st=0.1s sc=0.42908748175401623, sg=1)
[2021-04-14 20:02:22,201] [34616/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 20:02:22,201] [34616/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 3: {'alpha': 1.0}
[2021-04-14 20:02:22,202] [34616/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 20:02:22,202] [34616/Thread-2:local-0] [INFO] [dataiku.doctor.crossval.search_context] Fit s=2: alpha=1.0 .......................................................
[2021-04-14 20:02:22,769] [34616/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Done s=0: alpha=1.0 (ft=0.8s st=0.1s sc=0.6694766752306953, sg=1)
[2021-04-14 20:02:22,787] [34616/Thread-2:local-0] [INFO] [dataiku.doctor.crossval.search_context] Done s=2: alpha=1.0 (ft=0.5s st=0.1s sc=0.6738472683255519, sg=1)
[2021-04-14 20:02:22,803] [34616/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 20:02:22,804] [34616/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 20:02:22,805] [34616/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Done s=1: alpha=1.0 (ft=0.8s st=0.1s sc=0.6759940898897108, sg=1)
[2021-04-14 20:02:22,817] [34616/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 4: {'alpha': 1.0}
[2021-04-14 20:02:22,818] [34616/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 20:02:22,818] [34616/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 20:02:22,818] [34616/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Fit s=4: alpha=1.0 .......................................................
[2021-04-14 20:02:22,818] [34616/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 20:02:22,820] [34616/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Done s=4: alpha=3.0 (ft=0.8s st=0.1s sc=0.4297451224713804, sg=1)
[2021-04-14 20:02:22,825] [34616/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Fit s=3: alpha=1.0 .......................................................
[2021-04-14 20:02:22,841] [34616/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 20:02:23,275] [34616/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Done s=3: alpha=1.0 (ft=0.4s st=0.0s sc=0.6695676145042322, sg=1)
[2021-04-14 20:02:23,288] [34616/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 20:02:23,302] [34616/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Done s=4: alpha=1.0 (ft=0.4s st=0.0s sc=0.6709521607598197, sg=1)
[2021-04-14 20:02:23,315] [34616/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 20:02:23,975] [34616/Thread-1] [INFO] [dataiku.doctor.crossval.search_runner] Completed search for hyperparameters
[2021-04-14 20:02:23,975] [34616/Thread-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Scheduler has been soft interrupted
[2021-04-14 20:02:23,975] [34616/MainThread] [INFO] [dataiku.doctor.distributed.work_scheduler] Scheduler has been hard interrupted (shutdown)
[2021-04-14 20:02:23,975] [34616/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.local_worker] Stopping worker: local-1
[2021-04-14 20:02:23,975] [34616/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.local_worker] Stopped worker: local-1
[2021-04-14 20:02:23,975] [34616/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.local_worker] Stopping worker: local-2
[2021-04-14 20:02:23,976] [34616/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.local_worker] Stopped worker: local-2
[2021-04-14 20:02:23,976] [34616/Thread-5:local-3] [ERROR] [dataiku.doctor.distributed.work_scheduler] All workers are dead, interrupt the scheduler
[2021-04-14 20:02:23,976] [34616/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.local_worker] Stopping worker: local-0
[2021-04-14 20:02:23,976] [34616/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.local_worker] Stopped worker: local-0
[2021-04-14 20:02:23,976] [34616/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.local_worker] Stopping worker: local-3
[2021-04-14 20:02:23,976] [34616/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.local_worker] Stopped worker: local-3
[2021-04-14 20:02:23,976] [34616/MainThread] [INFO] [dataiku.doctor.crossval.search_runner] Hyperparameter search done, best_parameters being : {'alpha': 0.1}
[2021-04-14 20:02:23,976] [34616/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Hyperparameter searching
[2021-04-14 20:02:23,977] [34616/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Fitting model
[2021-04-14 20:02:23,977] [34616/MainThread] [INFO] [dataiku.doctor.prediction.common] Fitting model:
[2021-04-14 20:02:23,977] [34616/MainThread] [INFO] [dataiku.doctor.prediction.common]   Model is: Ridge(alpha=0.1, copy_X=True, fit_intercept=True, max_iter=None,
   normalize=True, random_state=None, solver='auto', tol=0.001)
[2021-04-14 20:02:23,977] [34616/MainThread] [INFO] [dataiku.doctor.prediction.common]   train_X class: <class 'numpy.ndarray'>
[2021-04-14 20:02:23,977] [34616/MainThread] [INFO] [dataiku.doctor.prediction.common]   train_X shape: (79912, 222)
[2021-04-14 20:02:23,977] [34616/MainThread] [INFO] [dataiku.doctor.prediction.common]   train_y shape: (79912,)
[2021-04-14 20:02:23,977] [34616/MainThread] [INFO] [dataiku.doctor.prediction.common]   calibration enabled: a sub-sample of the train data has been saved for calibration
[2021-04-14 20:02:24,200] [34616/MainThread] [INFO] [dataiku.doctor.prediction.regression_fit] Output params are {'resolved': {'algorithm': 'RIDGE_REGRESSION', 'skipExpensiveReports': False, 'ridge': {'alpha': 0.1}}, 'other': {}}
[2021-04-14 20:02:24,200] [34616/MainThread] [INFO] [dataiku.doctor.prediction.scoring_base] No feature importance in classifier
[2021-04-14 20:02:24,201] [34616/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Fitting model
[2021-04-14 20:02:24,201] [34616/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Saving model
[2021-04-14 20:02:24,249] [34616/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Saving model
[2021-04-14 20:02:24,249] [34616/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Scoring model
[2021-04-14 20:02:24,249] [34616/MainThread] [INFO] [dataiku.doctor.prediction.regression_scoring] Intrinsic scoring
[2021-04-14 20:02:24,249] [34616/MainThread] [INFO] [dataiku.doctor.prediction.regression_scoring] Extracting rescalers
[2021-04-14 20:02:24,249] [34616/MainThread] [INFO] [dataiku.doctor.prediction.scoring_base] Computing feature importance
[2021-04-14 20:02:24,249] [34616/MainThread] [INFO] [dataiku.doctor.prediction.scoring_base] No feature importance in classifier
[2021-04-14 20:02:24,250] [34616/MainThread] [INFO] [dataiku.doctor.prediction.regression_scoring] Computing regression coeffs
[2021-04-14 20:02:24,390] [34616/MainThread] [INFO] [dataiku.doctor.prediction.scoring_base] Coefs: [ 2.51469332e-01  1.63509404e-03  2.21563661e-04 -1.08438658e-04
  7.74309566e-04 -2.14135594e-03 -2.34731325e-03  4.05513302e-04
 -3.84224483e-03  9.98158460e-02  9.73352511e-03  3.31158614e-02
 -2.11510611e-02 -5.19063094e-02 -3.50484037e-03  7.58730492e-02
  1.87917073e-02  8.09058303e-03  1.07808991e-01 -1.42025066e-03
 -3.57123318e-02  8.00747399e-03  1.66052964e-02  1.25769508e-01
  3.87993505e-02  1.02901567e-01 -1.64331156e-02  2.74977094e-02
 -1.08664556e-02 -3.45798688e-02  9.45740161e-02  3.84245341e-02
  4.07675482e-02 -7.42267692e-02 -3.20179030e-02  3.11095285e-02
 -5.55087529e-02 -1.26314355e-01  3.40451396e-02 -8.33427970e-02
  6.34162347e-03 -6.44124595e-02 -6.80267451e-02 -1.32847284e-02
 -1.43999553e-02 -2.19698199e-02 -7.09643456e-02 -8.18525041e-02
 -5.25408099e-02 -4.11485849e-02 -1.83727132e-02  5.17117170e-02
 -5.09081170e-02 -3.42249204e-02 -7.03537148e-02 -8.67745154e-03
 -3.42042142e-02 -8.35265271e-02 -5.80790945e-03  1.31546007e-02
  1.52431821e-05 -7.14909469e-02 -1.76890406e-02 -6.18477912e-02
  2.93530349e-02 -4.81405730e-02 -6.29962315e-02 -2.53277467e-02
 -1.22505796e-01 -3.71036993e-02 -2.98264254e-03 -4.39029431e-02
 -4.32659362e-02 -4.08451428e-02 -2.35020194e-02 -3.80725572e-02
 -1.00058510e-02 -8.49418930e-02  7.68226921e-02  1.91134938e-02
 -5.51348656e-02 -2.60788252e-02 -1.26555785e-02 -1.10936365e-02
 -5.09018549e-02 -1.12611236e-02  9.65819090e-03 -1.18088341e-01
 -9.06406762e-02 -3.15950205e-02 -8.76668154e-02 -3.94059781e-02
 -7.21306777e-02 -5.47178272e-03 -1.17713620e-01 -7.19217366e-03
 -2.40140346e-02 -4.26262830e-02  1.09233151e-01 -3.88513090e-02
 -3.01049878e-02 -9.12018091e-03  1.76993949e-02 -4.33849341e-02
 -1.50910624e-02  5.36356829e-02 -8.30521621e-02 -9.53223882e-02
 -9.20811049e-03  0.00000000e+00 -3.24684595e-02 -5.56575564e-03
  0.00000000e+00  4.75576909e-03  4.32262494e-03 -2.86530591e-03
 -4.64573117e-03 -9.32227539e-03 -6.89112886e-04 -4.95519343e-03
  2.26771083e-03 -3.13123599e-03 -2.32389800e-03  8.18470544e-04
  4.66506757e-03 -1.37985824e-02 -4.01609863e-03  4.24942309e-03
 -7.24107016e-04  2.87219361e-03  5.66910899e-03 -1.88568513e-03
  1.54078413e-03  3.64628176e-03  1.48072790e-03  2.62869307e-03
  4.20827592e-03 -1.07439927e-03  1.24574167e-03  3.15160083e-03
  1.35274917e-03  5.18687989e-03  1.52483718e-03  4.68810540e-03
 -3.24950307e-03 -4.64729168e-02  1.84114868e-03 -3.73831883e-03
  5.92466517e-03  5.15693211e-03  5.96984770e-04 -2.21766570e-03
 -3.66906822e-03 -2.29490588e-03  1.52302188e-03  8.18906536e-04
 -3.87318318e-03  2.28162375e-03  5.88795753e-03  1.72018987e-03
 -3.29947956e-03  3.64051904e-03  2.81195950e-03  7.10490409e-03
 -1.25385160e-03  5.90594337e-03  2.95074233e-03  6.43181124e-03
 -5.64077423e-04  7.61913752e-03 -2.19629505e-02  6.73559783e-03
  1.98083604e-03 -2.19535312e-03  2.16464275e-03  4.31880406e-03
  4.88183979e-04  5.28669474e-03  2.08017592e-03 -2.42593590e-03
  9.06516547e-03  3.14693881e-03  9.34723871e-04  7.01639450e-03
  6.29829625e-03  8.48767241e-03 -2.68895548e-02  2.57829856e-03
  2.85306587e-03 -2.22267703e-02 -3.73465546e-03  5.98110690e-03
  5.70350899e-03  1.77794837e-03 -1.78132240e-03  2.07376526e-04
  6.07618794e-03 -1.83115172e-03  2.08935910e-03  4.04539004e-03
 -4.07757400e-05  4.06332876e-03 -6.82279558e-04  5.27112271e-03
  5.83756884e-03  6.96279724e-04  4.45258371e-03  1.99413364e-03
  3.68748616e-03  4.03704078e-03  2.83384309e-03  6.47040973e-03
  9.93450521e-03  2.59075846e-03  0.00000000e+00  1.88719506e-03
 -4.23056143e-03 -7.18683236e-04 -2.19331330e-03 -1.14283550e-02
  3.40048855e-05  7.52671568e-04  0.00000000e+00]
[2021-04-14 20:02:24,391] [34616/MainThread] [INFO] [dataiku.doctor.prediction.scoring_base] Sample variance: 0.0006883796472037912
[2021-04-14 20:02:24,533] [34616/MainThread] [INFO] [dataiku.doctor.prediction.scoring_base] Singular variance matrix
[2021-04-14 20:02:24,551] [34616/MainThread] [INFO] [dataiku.doctor.prediction.common] prepare multiframe shape=(20088,222) tn=4459536 nnz=206420 fill_ratio=0.05
[2021-04-14 20:02:24,551] [34616/MainThread] [INFO] [dataiku.doctor.prediction.common] too small, using array
[2021-04-14 20:02:24,591] [34616/MainThread] [INFO] [dataiku.doctor.prediction.regression_scoring] Creating predictions on test set
[2021-04-14 20:02:24,599] [34616/MainThread] [INFO] [dataiku.doctor.prediction.regression_scoring] Computing regression performance on [0.24861573 0.24861573 0.24861573 ... 0.22897359 0.23421346 0.24112227]
[2021-04-14 20:02:24,865] [34616/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Scoring model
[2021-04-14 20:02:24,866] [34616/MainThread] [INFO] [root] Reading with dtypes: None
[2021-04-14 20:02:24,866] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_month: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 20:02:24,866] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_woy: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 20:02:24,866] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_dow: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 20:02:24,866] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_hour: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 20:02:24,866] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for UniqueCarrier: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 20:02:24,866] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for FlightNum: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 20:02:24,866] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for AirTime: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 20:02:24,866] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Late: str (schema_type=boolean feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 20:02:24,866] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Origin: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 20:02:24,866] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Dest: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 20:02:24,866] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Distance: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 20:02:24,866] [34616/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Late_avg: <class 'numpy.float64'> (schema_type=double feature_type=NUMERIC feature_role=TARGET)
[2021-04-14 20:02:24,866] [34616/MainThread] [INFO] [root] Reading with FIXED dtypes: {'dep_month': <class 'numpy.float64'>, 'dep_woy': <class 'numpy.float64'>, 'dep_dow': <class 'numpy.float64'>, 'dep_hour': <class 'numpy.float64'>, 'UniqueCarrier': 'str', 'FlightNum': <class 'numpy.float64'>, 'AirTime': <class 'numpy.float64'>, 'Late': 'str', 'Origin': 'str', 'Dest': 'str', 'Distance': <class 'numpy.float64'>, 'Late_avg': <class 'numpy.float64'>}
[2021-04-14 20:02:24,894] [34616/MainThread] [INFO] [root] Loaded table
[2021-04-14 20:02:24,896] [34616/MainThread] [INFO] [dataiku.doctor.prediction.background_rows_handler] Building background rows
[2021-04-14 20:02:24,897] [34616/MainThread] [INFO] [dataiku.doctor.prediction.background_rows_handler] Background estimated bias <= 0.0007567449862710129
[2021-04-14 20:02:24,900] [34616/MainThread] [INFO] [dataiku.doctor.prediction.background_rows_handler] Built background rows with shape=(1000, 12)
[2021-04-14 20:02:24,900] [34616/MainThread] [INFO] [dataiku.doctor.prediction.background_rows_handler] Saving background rows with shape=(1000, 12)
[2021-04-14 20:02:24,908] [34616/MainThread] [INFO] [dataiku.doctor.utils]  Coercion done
[2021-04-14 20:02:24,909] [34616/MainThread] [INFO] [dataiku.doctor.prediction.histogram_handler] Computing histograms
[2021-04-14 20:02:25,129] [34616/MainThread] [INFO] [dataiku.doctor.prediction.histogram_handler] Saving histograms
[2021-04-14 20:02:25,157] [34616/MainThread] [INFO] [dataiku.doctor.prediction.column_importance_handler] Computing column importance
/data/home/dkuacademy/Applications/dataiku-dss-9.0.0/python/dataiku/doctor/prediction/column_importance_handler.py:132: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  clf.fit(preprocessed_array[idx], target[idx])
[2021-04-14 20:02:25,451] [34616/MainThread] [INFO] [dataiku.doctor.prediction.column_importance_handler] Saving column importance
[2021-04-14 20:02:25,483] [34616/MainThread] [INFO] [dataiku.base.socket_block_link] Client closed
[2021/04/14-20:02:25.483] [MRT-1906] [INFO] [dku.block.link.interaction]  - Check result for nullity exceptionIfNull=true result=not null
[2021/04/14-20:02:25.483] [MRT-1906] [INFO] [dku.analysis.prediction]  - Training returned ok
[2021/04/14-20:02:25.484] [MRT-1906] [INFO] [dku.ml.versioning]  - Dumping version info {"trainedWithDSSVersion":"9.0.0","trainedWithDSSConfVersion":"9000","trainedWithDSSBackend":"PY_MEMORY","backendCompatibilityVersion":18} in /data/home/dkuacademy/Library/dss-9.0/design/analysis-data/QS_DATAENG2/oSylVxXB/0IfwLWGQ/sessions/s8/pp1/m1
[2021/04/14-20:02:25.486] [MRT-1906] [INFO] [dku.analysis.ml]  - preTrain={
  "algorithm": "RIDGE_REGRESSION",
  "ridge_grid": {
    "alphaMode": "MANUAL",
    "alpha": {
      "gridMode": "EXPLICIT",
      "randomMode": "RANGE",
      "limit": {
        "min": 1.0E-23
      },
      "values": [
        0.1,
        1.0,
        3.0
      ],
      "range": {
        "min": 0.1,
        "max": 3.0,
        "nbValues": 3,
        "scaling": "LOGARITHMIC"
      }
    },
    "enabled": true
  },
  "max_ensemble_nodes_serialized": 0,
  "metrics": {
    "evaluationMetric": "R2",
    "customEvaluationMetricGIB": true,
    "customEvaluationMetricNeedsProba": false,
    "costMatrixWeights": {
      "tpGain": 1.0,
      "tnGain": 0.0,
      "fpGain": -0.3,
      "fnGain": 0.0
    },
    "liftPoint": 0.4
  },
  "autoOptimizeThreshold": true,
  "forcedClassifierThreshold": 0.0,
  "gridLength": 3,
  "grid_search_params": {
    "mode": "KFOLD",
    "splitRatio": 0.8,
    "shuffleIterations": 1,
    "nFolds": 5,
    "stratified": true,
    "strategy": "GRID",
    "bayesianOptimizer": "SCIKIT_OPTIMIZE",
    "randomized": true,
    "seed": 0,
    "nIter": 0,
    "nIterRandom": 3,
    "timeout": 0,
    "nJobs": 4,
    "distributed": false,
    "nContainers": 4
  },
  "pluginAlgoCustomGridSearch": false,
  "computeLearningCurves": false,
  "skipExpensiveReports": false
}
[2021/04/14-20:02:25.486] [MRT-1906] [INFO] [dku.analysis.ml]  - amp={
  "resolved": {
    "algorithm": "RIDGE_REGRESSION",
    "ridge": {
      "alpha": 0.1
    },
    "skipExpensiveReports": false
  },
  "other": {}
}
[2021/04/14-20:02:25.486] [MRT-1906] [WARN] [dku.kernels]  - Killing kernel python-single-command-kernel
[2021/04/14-20:02:25.487] [MRT-1906] [INFO] [dku.security.process]  - Killing process PID: 34616
[2021/04/14-20:02:25.490] [MRT-1906] [INFO] [dip.tickets]  - Destroying API ticket for analysis-ml-QS_DATAENG2-aU1av0J on behalf of tara
[2021/04/14-20:02:25.529] [KNL-python-single-command-kernel-monitor-1909] [INFO] [dku.kernels]  - Process done with code 143
[2021/04/14-20:02:25.533] [KNL-python-single-command-kernel-err-1912] [DEBUG] [process]  - StreamToLine: EOF (stream closed)
[2021/04/14-20:02:25.533] [KNL-python-single-command-kernel-out-1911] [DEBUG] [process]  - StreamToLine: EOF (stream closed)
[2021/04/14-20:02:25.533] [KNL-python-single-command-kernel-monitor-1909] [WARN] [dku.resource]  - stat file for pid 34616 does not exist. Process died?
[2021/04/14-20:02:25.534] [KNL-python-single-command-kernel-monitor-1909] [INFO] [dku.resourceusage]  - Reporting completion of CRU:{"context":{"type":"ANALYSIS_ML_TRAIN","authIdentifier":"tara","projectKey":"QS_DATAENG2","analysisId":"oSylVxXB","mlTaskId":"0IfwLWGQ","sessionId":"s8"},"type":"LOCAL_PROCESS","id":"fpYbjO7yDnRl2JsM","startTime":1618423336367,"localProcess":{"pid":34616,"commandName":"/data/home/dkuacademy/Library/dss-9.0/design/bin/python","cpuUserTimeMS":5630,"cpuSystemTimeMS":4430,"cpuChildrenUserTimeMS":0,"cpuChildrenSystemTimeMS":0,"cpuTotalMS":10060,"cpuCurrent":2.011597680463907,"vmSizeMB":3635,"vmRSSMB":1283,"vmHWMMB":1284,"vmRSSAnonMB":1263,"vmDataMB":2976,"vmSizePeakMB":3672,"vmRSSPeakMB":1283,"vmRSSTotalMBS":6415,"majorFaults":97,"childrenMajorFaults":0}}
[2021/04/14-20:02:25.534] [MRT-1906] [INFO] [dku.block.link]  - Closed socket
[2021/04/14-20:02:25.534] [MRT-1906] [INFO] [dku.block.link]  - Closed socket
[2021/04/14-20:02:25.534] [MRT-1906] [INFO] [dku.block.link]  - Closed serverSocket
[2021/04/14-20:02:25.535] [FT-TrainWorkThread-kfX1SEa1-1904] [INFO] [dku.analysis.ml.python] T-0IfwLWGQ - Processing thread joined ...
[2021/04/14-20:02:25.537] [FT-TrainWorkThread-kfX1SEa1-1904] [INFO] [dku.analysis.prediction] T-0IfwLWGQ - Train done
[2021/04/14-20:02:25.537] [FT-TrainWorkThread-kfX1SEa1-1904] [INFO] [dku.analysis.prediction] T-0IfwLWGQ - Train done
[2021/04/14-20:02:25.541] [FT-TrainWorkThread-kfX1SEa1-1904] [INFO] [dku.analysis.trainingdetails] T-0IfwLWGQ - Publishing mltask-train-done reflected event
