[2021/04/14-16:53:06.422] [FT-TrainWorkThread-4VpvYhxi-280193] [INFO] [dku.analysis.prediction]  - ******************************************
[2021/04/14-16:53:06.422] [FT-TrainWorkThread-4VpvYhxi-280193] [INFO] [dku.analysis.prediction]  - ** Start train session s4
[2021/04/14-16:53:06.422] [FT-TrainWorkThread-4VpvYhxi-280193] [INFO] [dku.analysis.prediction]  - ******************************************
[2021/04/14-16:53:06.423] [FT-TrainWorkThread-4VpvYhxi-280193] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 1] Search for split: p=type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337 i=bf10335945b1f843a4ae9838d57674b3-0
[2021/04/14-16:53:06.423] [FT-TrainWorkThread-4VpvYhxi-280193] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 1] Checking if splits are up to date. Policy: type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337, instance id: bf10335945b1f843a4ae9838d57674b3-0
[2021/04/14-16:53:06.424] [FT-TrainWorkThread-4VpvYhxi-280193] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 2] Search for split: p=type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337 i=bf10335945b1f843a4ae9838d57674b3-0
[2021/04/14-16:53:06.424] [FT-TrainWorkThread-4VpvYhxi-280193] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 2] Search for split: p=type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337 i=bf10335945b1f843a4ae9838d57674b3-0
[2021/04/14-16:53:06.425] [FT-TrainWorkThread-4VpvYhxi-280193] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 3] Checking if splits are up to date. Policy: type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337, instance id: bf10335945b1f843a4ae9838d57674b3-0
[2021/04/14-16:53:06.425] [FT-TrainWorkThread-4VpvYhxi-280193] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 3] Search for split: p=type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337 i=bf10335945b1f843a4ae9838d57674b3-0
[2021/04/14-16:53:06.426] [FT-TrainWorkThread-4VpvYhxi-280193] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 4] Search for split: p=type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337 i=bf10335945b1f843a4ae9838d57674b3-0
[2021/04/14-16:53:06.426] [FT-TrainWorkThread-4VpvYhxi-280193] [INFO] [dku.analysis.ml.python] T-0IfwLWGQ - Joining processing thread ...
[2021/04/14-16:53:06.427] [MRT-280194] [INFO] [dku.analysis.ml.python]  - Running a preprocessing set: pp1 in /data/home/dkuacademy/Library/dss-9.0/design/analysis-data/QS_DATAENG2/oSylVxXB/0IfwLWGQ/sessions/s4/pp1
[2021/04/14-16:53:06.427] [MRT-280195] [INFO] [dku.analysis.ml.python]  - TrainAdditionalThread done
[2021/04/14-16:53:06.427] [MRT-280194] [INFO] [dku.block.link]  - Started a socket on port 44907
[2021/04/14-16:53:06.428] [MRT-280194] [INFO] [dku.ml.kernel]  - Writing output of python-single-command-kernel to /data/home/dkuacademy/Library/dss-9.0/design/analysis-data/QS_DATAENG2/oSylVxXB/0IfwLWGQ/sessions/s4/pp1/train.log
[2021/04/14-16:53:06.428] [MRT-280194] [INFO] [dku.code.envs.resolution]  - Executing Python activity in builtin env
[2021/04/14-16:53:06.428] [MRT-280194] [WARN] [dku.code.projectLibs]  - External libraries file not found: /data/home/dkuacademy/Library/dss-9.0/design/config/projects/QS_DATAENG2/lib/external-libraries.json
[2021/04/14-16:53:06.428] [MRT-280194] [INFO] [dku.code.projectLibs]  - EXTERNAL LIBS FROM QS_DATAENG2 is {"gitReferences":{},"pythonPath":["python"],"rsrcPath":["R"],"importLibrariesFromProjects":[]}
[2021/04/14-16:53:06.429] [MRT-280194] [INFO] [dku.code.projectLibs]  - chunkFolder is /data/home/dkuacademy/Library/dss-9.0/design/config/projects/QS_DATAENG2/lib/R
[2021/04/14-16:53:06.429] [MRT-280194] [INFO] [dku.python.single_command.kernel]  - Starting Python process for kernel  python-single-command-kernel
[2021/04/14-16:53:06.429] [MRT-280194] [INFO] [dip.tickets]  - Creating API ticket for analysis-ml-QS_DATAENG2-1d2hoIW on behalf of tara id=analysis-ml-QS_DATAENG2-1d2hoIW_1sOAZG39LegQ
[2021/04/14-16:53:06.429] [MRT-280194] [INFO] [dku.security.process]  - Starting process (regular)
[2021/04/14-16:53:06.431] [MRT-280194] [INFO] [dku.security.process]  - Process started with pid=482260
[2021/04/14-16:53:06.431] [MRT-280194] [INFO] [dku.processes.cgroups]  - Will use cgroups []
[2021/04/14-16:53:06.431] [MRT-280194] [INFO] [dku.processes.cgroups]  - Applying rules to used cgroups: []
[2021/04/14-16:53:06.434] [KNL-python-single-command-kernel-monitor-280198] [INFO] [dku.resourceusage]  - Reporting start of CRU:{"context":{"type":"ANALYSIS_ML_TRAIN","authIdentifier":"tara","projectKey":"QS_DATAENG2","analysisId":"oSylVxXB","mlTaskId":"0IfwLWGQ","sessionId":"s4"},"type":"LOCAL_PROCESS","id":"WgxeZ7Ve8u3KpLMh","startTime":1618411986434,"localProcess":{"cpuCurrent":0.0}}
[2021/04/14-16:53:06.435] [process-resource-monitor-482260-280199] [DEBUG] [dku.resource]  - Process stats for pid 482260: {"pid":482260,"commandName":"/data/home/dkuacademy/Library/dss-9.0/design/bin/python","cpuUserTimeMS":0,"cpuSystemTimeMS":0,"cpuChildrenUserTimeMS":0,"cpuChildrenSystemTimeMS":0,"cpuTotalMS":0,"cpuCurrent":0.0,"vmSizeMB":10,"vmRSSMB":0,"vmHWMMB":0,"vmRSSAnonMB":0,"vmDataMB":0,"vmSizePeakMB":10,"vmRSSPeakMB":0,"vmRSSTotalMBS":0,"majorFaults":0,"childrenMajorFaults":0}
Installing debugging signal handler
[2021-04-14 16:53:12,823] [482260/MainThread] [INFO] [dataiku.base.socket_block_link] Connecting to localhost (127.0.0.1) at port 44907
[2021-04-14 16:53:12,823] [482260/MainThread] [INFO] [dataiku.base.socket_block_link] Connected to localhost (127.0.0.1) at port 44907
[2021/04/14-16:53:12.826] [MRT-280194] [INFO] [dku.link.secret_protected]  - Connected to kernel
[2021/04/14-16:53:12.827] [MRT-280194] [INFO] [dku.block.link.interaction]  - Execute link command respClazz=true respTypeToken=false respIsString=false is=false asyncInputStream=false os=false
[2021-04-14 16:53:18,403] [482260/MainThread] [INFO] [dataiku.doctor.utils.dku_pickle] Setting cloudpickle as the pickling tool
[2021-04-14 16:53:18,421] [482260/MainThread] [INFO] [root] Running analysis command: train_prediction_models_nosave
[2021-04-14 16:53:18,421] [482260/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: DatasetSanityCheckDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_DATASET_SANITY_CHECKS
[2021-04-14 16:53:18,421] [482260/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: ClassifierAccuracyCheckDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_MODEL_CHECK
[2021-04-14 16:53:18,421] [482260/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: RegressionR2CheckDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_MODEL_CHECK
[2021-04-14 16:53:18,421] [482260/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: LeakageDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_LEAKAGE_DETECTION
[2021-04-14 16:53:18,421] [482260/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: TreeOverfitDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_TRAINING_OVERFIT
[2021-04-14 16:53:18,421] [482260/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: MLAssertionsDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_ML_ASSERTIONS
[2021-04-14 16:53:18,421] [482260/MainThread] [INFO] [dataiku.doctor.commands] PPS is {u'preprocessingFitSampleSeed': 1337, u'feature_selection_params': {u'custom_params': {u'code': u'# type your code here'}, u'pca_params': {u'variance_proportion': 0.9, u'n_features': 25}, u'random_forest_params': {u'depth': 10, u'n_features': 25, u'n_trees': 30}, u'lasso_params': {u'alpha': [0.01, 0.1, 1.0, 10.0, 100.0], u'cross_validate': True}, u'method': u'NONE', u'correlation_params': {u'max_abs_correlation': 1.0, u'n_features': 25, u'min_abs_correlation': 0.0}}, u'preprocessingFitSampleRatio': 1.0, u'reduce': {u'enabled': False, u'kept_variance': 0.0}, u'skipPreprocessing': False, u'target_remapping': [], u'per_feature': {u'Origin': {u'missing_impute_with': u'MODE', u'autoReason': u'REJECT_CARDINALITY', u'hash_whole_categories': True, u'nb_bins_hashing': 1048576, u'customHandlingCode': u'', u'category_handling': u'DUMMIFY', u'cumulative_proportion': 0.95, u'sendToInput': u'main', u'state': {u'userModified': False, u'recordedMeaning': u'Text', u'autoModifiedByDSS': False}, u'missing_handling': u'NONE', u'max_nb_categories': 100, u'customProcessorWantsMatrix': False, u'dummy_clip': u'MAX_NB_CATEGORIES', u'role': u'INPUT', u'max_cat_safety': 200, u'min_samples': 10, u'type': u'CATEGORY', u'dummy_drop': u'NONE'}, u'dep_hour': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'UniqueCarrier': {u'missing_impute_with': u'MODE', u'customProcessorWantsMatrix': False, u'hash_whole_categories': True, u'nb_bins_hashing': 1048576, u'customHandlingCode': u'', u'category_handling': u'DUMMIFY', u'cumulative_proportion': 0.95, u'sendToInput': u'main', u'state': {u'userModified': False, u'recordedMeaning': u'Text', u'autoModifiedByDSS': False}, u'missing_handling': u'NONE', u'max_nb_categories': 5, u'dummy_clip': u'MAX_NB_CATEGORIES', u'role': u'INPUT', u'max_cat_safety': 200, u'min_samples': 10, u'type': u'CATEGORY', u'dummy_drop': u'DROP'}, u'FlightNum': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'Late_avg': {u'generate_derivative': False, u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'sendToInput': u'main', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'DoubleMeaning', u'autoModifiedByDSS': False}, u'role': u'TARGET', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'dep_woy': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'Distance': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'Dest': {u'missing_impute_with': u'MODE', u'autoReason': u'REJECT_CARDINALITY', u'hash_whole_categories': True, u'nb_bins_hashing': 1048576, u'customHandlingCode': u'', u'category_handling': u'DUMMIFY', u'cumulative_proportion': 0.95, u'sendToInput': u'main', u'state': {u'userModified': False, u'recordedMeaning': u'Text', u'autoModifiedByDSS': False}, u'missing_handling': u'NONE', u'max_nb_categories': 100, u'customProcessorWantsMatrix': False, u'dummy_clip': u'MAX_NB_CATEGORIES', u'role': u'INPUT', u'max_cat_safety': 200, u'min_samples': 10, u'type': u'CATEGORY', u'dummy_drop': u'NONE'}, u'dep_month': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'dep_dow': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'Late': {u'missing_impute_with': u'MODE', u'customProcessorWantsMatrix': False, u'hash_whole_categories': True, u'nb_bins_hashing': 1048576, u'customHandlingCode': u'', u'category_handling': u'DUMMIFY', u'cumulative_proportion': 0.95, u'sendToInput': u'main', u'state': {u'userModified': False, u'recordedMeaning': u'Boolean', u'autoModifiedByDSS': False}, u'missing_handling': u'NONE', u'max_nb_categories': 5, u'dummy_clip': u'MAX_NB_CATEGORIES', u'role': u'INPUT', u'max_cat_safety': 200, u'min_samples': 10, u'type': u'CATEGORY', u'dummy_drop': u'DROP'}, u'AirTime': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}}, u'feature_generation': {u'manual_interactions': {u'interactions': []}, u'pairwise_linear': {u'behavior': u'DISABLED'}, u'categoricals_count_transformer': {u'input_features': [], u'all_features': False, u'behavior': u'DISABLED'}, u'polynomial_combinations': {u'behavior': u'DISABLED'}, u'numericals_clustering': {u'k': 0, u'input_features': [], u'all_features': False, u'behavior': u'DISABLED'}}}
[2021-04-14 16:53:18,421] [482260/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Loading train set
[2021-04-14 16:53:18,422] [482260/MainThread] [INFO] [root] Reading with dtypes: None
[2021-04-14 16:53:18,422] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_month: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 16:53:18,422] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_woy: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 16:53:18,422] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_dow: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 16:53:18,422] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_hour: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 16:53:18,422] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for UniqueCarrier: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 16:53:18,422] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for FlightNum: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 16:53:18,422] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for AirTime: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 16:53:18,422] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Late: str (schema_type=boolean feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 16:53:18,422] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Origin: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 16:53:18,422] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Dest: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 16:53:18,422] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Distance: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 16:53:18,422] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Late_avg: <type 'numpy.float64'> (schema_type=double feature_type=NUMERIC feature_role=TARGET)
[2021-04-14 16:53:18,422] [482260/MainThread] [INFO] [root] Reading with FIXED dtypes: {u'dep_hour': <type 'numpy.float64'>, u'Late': 'str', u'UniqueCarrier': 'str', u'FlightNum': <type 'numpy.float64'>, u'Late_avg': <type 'numpy.float64'>, u'dep_woy': <type 'numpy.float64'>, u'Distance': <type 'numpy.float64'>, u'Dest': 'str', u'Origin': 'str', u'dep_month': <type 'numpy.float64'>, u'dep_dow': <type 'numpy.float64'>, u'AirTime': <type 'numpy.float64'>}
[2021-04-14 16:53:18,539] [482260/MainThread] [INFO] [root] Loaded table
[2021-04-14 16:53:18,539] [482260/MainThread] [INFO] [dataiku.doctor.utils]  Coercion done
[2021-04-14 16:53:18,540] [482260/MainThread] [INFO] [dataiku.doctor.utils.split] Loaded train df: shape=(79912,12)
[2021-04-14 16:53:18,540] [482260/MainThread] [INFO] [dataiku.doctor.commands] Train col : dep_month (float64)
[2021-04-14 16:53:18,540] [482260/MainThread] [INFO] [dataiku.doctor.commands] Train col : dep_woy (float64)
[2021-04-14 16:53:18,540] [482260/MainThread] [INFO] [dataiku.doctor.commands] Train col : dep_dow (float64)
[2021-04-14 16:53:18,540] [482260/MainThread] [INFO] [dataiku.doctor.commands] Train col : dep_hour (float64)
[2021-04-14 16:53:18,540] [482260/MainThread] [INFO] [dataiku.doctor.commands] Train col : UniqueCarrier (object)
[2021-04-14 16:53:18,540] [482260/MainThread] [INFO] [dataiku.doctor.commands] Train col : FlightNum (float64)
[2021-04-14 16:53:18,540] [482260/MainThread] [INFO] [dataiku.doctor.commands] Train col : AirTime (float64)
[2021-04-14 16:53:18,540] [482260/MainThread] [INFO] [dataiku.doctor.commands] Train col : Late (object)
[2021-04-14 16:53:18,540] [482260/MainThread] [INFO] [dataiku.doctor.commands] Train col : Origin (object)
[2021-04-14 16:53:18,540] [482260/MainThread] [INFO] [dataiku.doctor.commands] Train col : Dest (object)
[2021-04-14 16:53:18,540] [482260/MainThread] [INFO] [dataiku.doctor.commands] Train col : Distance (float64)
[2021-04-14 16:53:18,540] [482260/MainThread] [INFO] [dataiku.doctor.commands] Train col : Late_avg (float64)
[2021-04-14 16:53:18,540] [482260/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Loading train set
[2021-04-14 16:53:18,540] [482260/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Loading test set
[2021-04-14 16:53:18,541] [482260/MainThread] [INFO] [root] Reading with dtypes: None
[2021-04-14 16:53:18,541] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_month: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 16:53:18,541] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_woy: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 16:53:18,541] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_dow: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 16:53:18,541] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_hour: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 16:53:18,541] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for UniqueCarrier: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 16:53:18,541] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for FlightNum: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 16:53:18,541] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for AirTime: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 16:53:18,541] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Late: str (schema_type=boolean feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 16:53:18,541] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Origin: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 16:53:18,541] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Dest: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 16:53:18,541] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Distance: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 16:53:18,541] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Late_avg: <type 'numpy.float64'> (schema_type=double feature_type=NUMERIC feature_role=TARGET)
[2021-04-14 16:53:18,541] [482260/MainThread] [INFO] [root] Reading with FIXED dtypes: {u'dep_hour': <type 'numpy.float64'>, u'Late': 'str', u'UniqueCarrier': 'str', u'FlightNum': <type 'numpy.float64'>, u'Late_avg': <type 'numpy.float64'>, u'dep_woy': <type 'numpy.float64'>, u'Distance': <type 'numpy.float64'>, u'Dest': 'str', u'Origin': 'str', u'dep_month': <type 'numpy.float64'>, u'dep_dow': <type 'numpy.float64'>, u'AirTime': <type 'numpy.float64'>}
[2021-04-14 16:53:18,572] [482260/MainThread] [INFO] [root] Loaded table
[2021-04-14 16:53:18,572] [482260/MainThread] [INFO] [dataiku.doctor.utils]  Coercion done
[2021-04-14 16:53:18,573] [482260/MainThread] [INFO] [dataiku.doctor.utils.split] Loaded test df: shape=(20088,12)
[2021-04-14 16:53:18,573] [482260/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Loading test set
[2021-04-14 16:53:18,573] [482260/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Collecting statistics
[2021-04-14 16:53:19,477] [482260/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Origin... (type=CATEGORY)
[2021-04-14 16:53:19,489] [482260/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at dep_hour... (type=NUMERIC)
[2021-04-14 16:53:19,490] [482260/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-14 16:53:19,493] [482260/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at UniqueCarrier... (type=CATEGORY)
[2021-04-14 16:53:19,505] [482260/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at FlightNum... (type=NUMERIC)
[2021-04-14 16:53:19,505] [482260/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-14 16:53:19,507] [482260/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Late_avg... (type=NUMERIC)
[2021-04-14 16:53:19,508] [482260/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at dep_woy... (type=NUMERIC)
[2021-04-14 16:53:19,508] [482260/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-14 16:53:19,510] [482260/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Distance... (type=NUMERIC)
[2021-04-14 16:53:19,510] [482260/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-14 16:53:19,513] [482260/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Dest... (type=CATEGORY)
[2021-04-14 16:53:19,524] [482260/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at dep_month... (type=NUMERIC)
[2021-04-14 16:53:19,524] [482260/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-14 16:53:19,527] [482260/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at dep_dow... (type=NUMERIC)
[2021-04-14 16:53:19,527] [482260/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-14 16:53:19,529] [482260/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Late... (type=CATEGORY)
[2021-04-14 16:53:19,541] [482260/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at AirTime... (type=NUMERIC)
[2021-04-14 16:53:19,541] [482260/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-14 16:53:19,544] [482260/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Collecting statistics
[2021-04-14 16:53:19,545] [482260/MainThread] [INFO] [dataiku.doctor.multiframe] generating interactions
[2021-04-14 16:53:19,545] [482260/MainThread] [INFO] [dataiku.doctor.multiframe] {u'preprocessingFitSampleSeed': 1337, u'feature_selection_params': {u'custom_params': {u'code': u'# type your code here'}, u'pca_params': {u'variance_proportion': 0.9, u'n_features': 25}, u'random_forest_params': {u'depth': 10, u'n_features': 25, u'n_trees': 30}, u'lasso_params': {u'alpha': [0.01, 0.1, 1.0, 10.0, 100.0], u'cross_validate': True}, u'method': u'NONE', u'correlation_params': {u'max_abs_correlation': 1.0, u'n_features': 25, u'min_abs_correlation': 0.0}}, u'preprocessingFitSampleRatio': 1.0, u'reduce': {u'enabled': False, u'kept_variance': 0.0}, u'skipPreprocessing': False, u'target_remapping': [], u'per_feature': {u'Origin': {u'missing_impute_with': u'MODE', u'autoReason': u'REJECT_CARDINALITY', u'hash_whole_categories': True, u'nb_bins_hashing': 1048576, u'customHandlingCode': u'', u'category_handling': u'DUMMIFY', u'cumulative_proportion': 0.95, u'sendToInput': u'main', u'state': {u'userModified': False, u'recordedMeaning': u'Text', u'autoModifiedByDSS': False}, u'missing_handling': u'NONE', u'max_nb_categories': 100, u'customProcessorWantsMatrix': False, u'dummy_clip': u'MAX_NB_CATEGORIES', u'role': u'INPUT', u'max_cat_safety': 200, u'min_samples': 10, u'type': u'CATEGORY', u'dummy_drop': u'NONE'}, u'dep_hour': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'UniqueCarrier': {u'missing_impute_with': u'MODE', u'customProcessorWantsMatrix': False, u'hash_whole_categories': True, u'nb_bins_hashing': 1048576, u'customHandlingCode': u'', u'category_handling': u'DUMMIFY', u'cumulative_proportion': 0.95, u'sendToInput': u'main', u'state': {u'userModified': False, u'recordedMeaning': u'Text', u'autoModifiedByDSS': False}, u'missing_handling': u'NONE', u'max_nb_categories': 5, u'dummy_clip': u'MAX_NB_CATEGORIES', u'role': u'INPUT', u'max_cat_safety': 200, u'min_samples': 10, u'type': u'CATEGORY', u'dummy_drop': u'DROP'}, u'FlightNum': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'Late_avg': {u'generate_derivative': False, u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'sendToInput': u'main', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'DoubleMeaning', u'autoModifiedByDSS': False}, u'role': u'TARGET', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'dep_woy': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'Distance': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'Dest': {u'missing_impute_with': u'MODE', u'autoReason': u'REJECT_CARDINALITY', u'hash_whole_categories': True, u'nb_bins_hashing': 1048576, u'customHandlingCode': u'', u'category_handling': u'DUMMIFY', u'cumulative_proportion': 0.95, u'sendToInput': u'main', u'state': {u'userModified': False, u'recordedMeaning': u'Text', u'autoModifiedByDSS': False}, u'missing_handling': u'NONE', u'max_nb_categories': 100, u'customProcessorWantsMatrix': False, u'dummy_clip': u'MAX_NB_CATEGORIES', u'role': u'INPUT', u'max_cat_safety': 200, u'min_samples': 10, u'type': u'CATEGORY', u'dummy_drop': u'NONE'}, u'dep_month': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'dep_dow': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'Late': {u'missing_impute_with': u'MODE', u'customProcessorWantsMatrix': False, u'hash_whole_categories': True, u'nb_bins_hashing': 1048576, u'customHandlingCode': u'', u'category_handling': u'DUMMIFY', u'cumulative_proportion': 0.95, u'sendToInput': u'main', u'state': {u'userModified': False, u'recordedMeaning': u'Boolean', u'autoModifiedByDSS': False}, u'missing_handling': u'NONE', u'max_nb_categories': 5, u'dummy_clip': u'MAX_NB_CATEGORIES', u'role': u'INPUT', u'max_cat_safety': 200, u'min_samples': 10, u'type': u'CATEGORY', u'dummy_drop': u'DROP'}, u'AirTime': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}}, u'feature_generation': {u'manual_interactions': {u'interactions': []}, u'pairwise_linear': {u'behavior': u'DISABLED'}, u'categoricals_count_transformer': {u'input_features': [], u'all_features': False, u'behavior': u'DISABLED'}, u'polynomial_combinations': {u'behavior': u'DISABLED'}, u'numericals_clustering': {u'k': 0, u'input_features': [], u'all_features': False, u'behavior': u'DISABLED'}}}
[2021-04-14 16:53:19,545] [482260/MainThread] [INFO] [dataiku.doctor.multiframe] No feature selection to perform
[2021-04-14 16:53:19,545] [482260/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Preprocessing train set
[2021-04-14 16:53:19,615] [482260/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 79912
[2021-04-14 16:53:19,615] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RemapValueToOutput
[2021-04-14 16:53:19,615] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:MultipleImputeMissingFromInput
[2021-04-14 16:53:19,615] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {u'dep_hour': 13.086732906196817, u'Distance': 724.2432175392933, u'FlightNum': 2149.660088597457, u'dep_woy': 14.544386324957452, u'dep_month': 3.7664931424567025, u'dep_dow': 3.9422364601061166, u'AirTime': 103.37577585343878}
[2021-04-14 16:53:19,621] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (dep_hour)
[2021-04-14 16:53:19,621] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_hour (avg=13.0867329062 std=4.80188247322 shift=13.0867329062 inv_scale=0.208251660797)
[2021-04-14 16:53:19,790] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_hour (avg=6.69312549233e-16 std=1.0) nulls=0
[2021-04-14 16:53:19,790] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (FlightNum)
[2021-04-14 16:53:19,791] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale FlightNum (avg=2149.6600886 std=1944.09274632 shift=2149.6600886 inv_scale=0.000514378751678)
[2021-04-14 16:53:19,792] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled FlightNum (avg=3.87140820508e-16 std=1.0) nulls=0
[2021-04-14 16:53:19,793] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (dep_woy)
[2021-04-14 16:53:19,793] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_woy (avg=14.544386325 std=6.97121381815 shift=14.544386325 inv_scale=0.143447041804)
[2021-04-14 16:53:19,795] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_woy (avg=2.01557300121e-13 std=1.0) nulls=0
[2021-04-14 16:53:19,795] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (Distance)
[2021-04-14 16:53:19,795] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale Distance (avg=724.243217539 std=561.810411687 shift=724.243217539 inv_scale=0.00177995989252)
[2021-04-14 16:53:19,797] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled Distance (avg=1.16862949168e-16 std=1.0) nulls=0
[2021-04-14 16:53:19,797] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (dep_month)
[2021-04-14 16:53:19,797] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_month (avg=3.76649314246 std=1.59191042442 shift=3.76649314246 inv_scale=0.628176048512)
[2021-04-14 16:53:19,799] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_month (avg=1.15324419462e-13 std=1.0) nulls=0
[2021-04-14 16:53:19,799] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (dep_dow)
[2021-04-14 16:53:19,800] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_dow (avg=3.94223646011 std=1.9826725071 shift=3.94223646011 inv_scale=0.50436973147)
[2021-04-14 16:53:19,802] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_dow (avg=1.69635498405e-15 std=1.0) nulls=0
[2021-04-14 16:53:19,802] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (AirTime)
[2021-04-14 16:53:19,802] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale AirTime (avg=103.375775853 std=68.4857135559 shift=103.375775853 inv_scale=0.0146015854706)
[2021-04-14 16:53:19,804] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled AirTime (avg=5.50103338372e-15 std=1.0) nulls=0
[2021-04-14 16:53:19,804] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FlushDFBuilder(num_flagonly)
[2021-04-14 16:53:19,804] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FastSparseDummifyProcessor (Origin)
[2021-04-14 16:53:19,821] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(79912, 103) nnz=79912
[2021-04-14 16:53:19,821] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FastSparseDummifyProcessor (UniqueCarrier)
[2021-04-14 16:53:19,835] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(79912, 7) nnz=41098
[2021-04-14 16:53:19,835] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FastSparseDummifyProcessor (Dest)
[2021-04-14 16:53:19,851] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(79912, 103) nnz=79912
[2021-04-14 16:53:19,852] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FastSparseDummifyProcessor (Late)
[2021-04-14 16:53:19,866] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(79912, 2) nnz=60354
[2021-04-14 16:53:19,867] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:MultipleImputeMissingFromInput
[2021-04-14 16:53:19,867] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2021-04-14 16:53:19,867] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FlushDFBuilder(cat_flagpresence)
[2021-04-14 16:53:19,867] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:MultipleImputeMissingFromInput
[2021-04-14 16:53:19,867] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2021-04-14 16:53:19,867] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FlushDFBuilder(interaction)
[2021-04-14 16:53:19,867] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RealignTarget
[2021-04-14 16:53:19,867] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] Realign target series = (79912,)
[2021-04-14 16:53:19,869] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] After realign target: (79912,)
[2021-04-14 16:53:19,869] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:DropRowsWhereNoTarget
[2021-04-14 16:53:19,870] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] Deleting 0 rows because one of ['target'] is missing
[2021-04-14 16:53:19,870] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] MF before = (79912, 222)
[2021-04-14 16:53:19,870] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] target before = (79912,)
[2021-04-14 16:53:19,875] [482260/MainThread] [INFO] [dataiku.doctor.multiframe] MultiFrame, dropping rows: []
[2021-04-14 16:53:19,898] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] After DRWNT input_df=(79912, 12)
[2021-04-14 16:53:19,898] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] MF after = (79912, 222)
[2021-04-14 16:53:19,898] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] target after = (79912,)
[2021-04-14 16:53:19,898] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:DumpPipelineState
[2021-04-14 16:53:19,898] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] ********* Pipeline state (Before feature selection)
[2021-04-14 16:53:19,898] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]    input_df= (79912, 12) 
[2021-04-14 16:53:19,899] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]    current_mf=(79912, 222) 
[2021-04-14 16:53:19,899] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]    PPR: 
[2021-04-14 16:53:19,899] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]       target = <class 'pandas.core.series.Series'> ((79912,))
[2021-04-14 16:53:19,899] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:EmitCurrentMFAsResult
[2021-04-14 16:53:19,899] [482260/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 79912
[2021-04-14 16:53:19,899] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:DumpPipelineState
[2021-04-14 16:53:19,899] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] ********* Pipeline state (At end)
[2021-04-14 16:53:19,899] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]    input_df= (79912, 12) 
[2021-04-14 16:53:19,899] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]    current_mf=(0, 0) 
[2021-04-14 16:53:19,899] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]    PPR: 
[2021-04-14 16:53:19,899] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]       UNPROCESSED = <class 'pandas.core.frame.DataFrame'> ((79912, 12))
[2021-04-14 16:53:19,899] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]       TRAIN = <class 'dataiku.doctor.multiframe.MultiFrame'> ((79912, 222))
[2021-04-14 16:53:19,899] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]       target = <class 'pandas.core.series.Series'> ((79912,))
[2021-04-14 16:53:19,900] [482260/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Preprocessing train set
[2021-04-14 16:53:19,900] [482260/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Preprocessing test set
[2021-04-14 16:53:20,122] [482260/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 20088
[2021-04-14 16:53:20,122] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RemapValueToOutput
[2021-04-14 16:53:20,122] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:MultipleImputeMissingFromInput
[2021-04-14 16:53:20,122] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {u'dep_hour': 13.086732906196817, u'Distance': 724.2432175392933, u'FlightNum': 2149.660088597457, u'dep_woy': 14.544386324957452, u'dep_month': 3.7664931424567025, u'dep_dow': 3.9422364601061166, u'AirTime': 103.37577585343878}
[2021-04-14 16:53:20,124] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (dep_hour)
[2021-04-14 16:53:20,124] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_hour (avg=13.1375448029 std=4.81152184405 shift=13.0867329062 inv_scale=0.208251660797)
[2021-04-14 16:53:20,125] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_hour (avg=0.0105816618699 std=1.00200741499) nulls=0
[2021-04-14 16:53:20,125] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (FlightNum)
[2021-04-14 16:53:20,126] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale FlightNum (avg=2129.26144962 std=1938.80005733 shift=2149.6600886 inv_scale=0.000514378751678)
[2021-04-14 16:53:20,127] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled FlightNum (avg=-0.0104926264523 std=0.997277553245) nulls=0
[2021-04-14 16:53:20,127] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (dep_woy)
[2021-04-14 16:53:20,127] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_woy (avg=14.5089107925 std=6.93786261177 shift=14.544386325 inv_scale=0.143447041804)
[2021-04-14 16:53:20,128] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_woy (avg=-0.00508886018562 std=0.995215868103) nulls=0
[2021-04-14 16:53:20,128] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (Distance)
[2021-04-14 16:53:20,128] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale Distance (avg=726.248058542 std=562.197650727 shift=724.243217539 inv_scale=0.00177995989252)
[2021-04-14 16:53:20,129] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled Distance (avg=0.00356853657642 std=1.00068926996) nulls=0
[2021-04-14 16:53:20,129] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (dep_month)
[2021-04-14 16:53:20,130] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_month (avg=3.75985663082 std=1.58189507051 shift=3.76649314246 inv_scale=0.628176048512)
[2021-04-14 16:53:20,131] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_month (avg=-0.00416889765302 std=0.993708594551) nulls=0
[2021-04-14 16:53:20,131] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (dep_dow)
[2021-04-14 16:53:20,131] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_dow (avg=3.92751891677 std=1.98353439579 shift=3.94223646011 inv_scale=0.50436973147)
[2021-04-14 16:53:20,132] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_dow (avg=-0.00742308338224 std=1.00043471057) nulls=0
[2021-04-14 16:53:20,132] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (AirTime)
[2021-04-14 16:53:20,132] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale AirTime (avg=103.513590203 std=68.678091524 shift=103.375775853 inv_scale=0.0146015854706)
[2021-04-14 16:53:20,133] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled AirTime (avg=0.00201230800574 std=1.00280902334) nulls=0
[2021-04-14 16:53:20,134] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FlushDFBuilder(num_flagonly)
[2021-04-14 16:53:20,134] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FastSparseDummifyProcessor (Origin)
[2021-04-14 16:53:20,139] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(20088, 103) nnz=20088
[2021-04-14 16:53:20,139] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FastSparseDummifyProcessor (UniqueCarrier)
[2021-04-14 16:53:20,143] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(20088, 7) nnz=10425
[2021-04-14 16:53:20,143] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FastSparseDummifyProcessor (Dest)
[2021-04-14 16:53:20,149] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(20088, 103) nnz=20088
[2021-04-14 16:53:20,149] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FastSparseDummifyProcessor (Late)
[2021-04-14 16:53:20,153] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(20088, 2) nnz=15203
[2021-04-14 16:53:20,154] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:MultipleImputeMissingFromInput
[2021-04-14 16:53:20,154] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2021-04-14 16:53:20,154] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FlushDFBuilder(cat_flagpresence)
[2021-04-14 16:53:20,154] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:MultipleImputeMissingFromInput
[2021-04-14 16:53:20,154] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2021-04-14 16:53:20,154] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FlushDFBuilder(interaction)
[2021-04-14 16:53:20,154] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RealignTarget
[2021-04-14 16:53:20,154] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] Realign target series = (20088,)
[2021-04-14 16:53:20,155] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] After realign target: (20088,)
[2021-04-14 16:53:20,155] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:DropRowsWhereNoTarget
[2021-04-14 16:53:20,155] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] Deleting 0 rows because one of ['target'] is missing
[2021-04-14 16:53:20,155] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] MF before = (20088, 222)
[2021-04-14 16:53:20,155] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] target before = (20088,)
[2021-04-14 16:53:20,157] [482260/MainThread] [INFO] [dataiku.doctor.multiframe] MultiFrame, dropping rows: []
[2021-04-14 16:53:20,164] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] After DRWNT input_df=(20088, 12)
[2021-04-14 16:53:20,164] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] MF after = (20088, 222)
[2021-04-14 16:53:20,164] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] target after = (20088,)
[2021-04-14 16:53:20,164] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:DumpPipelineState
[2021-04-14 16:53:20,165] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] ********* Pipeline state (Before feature selection)
[2021-04-14 16:53:20,165] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]    input_df= (20088, 12) 
[2021-04-14 16:53:20,165] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]    current_mf=(20088, 222) 
[2021-04-14 16:53:20,165] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]    PPR: 
[2021-04-14 16:53:20,165] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]       target = <class 'pandas.core.series.Series'> ((20088,))
[2021-04-14 16:53:20,165] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:EmitCurrentMFAsResult
[2021-04-14 16:53:20,165] [482260/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 20088
[2021-04-14 16:53:20,165] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:DumpPipelineState
[2021-04-14 16:53:20,165] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] ********* Pipeline state (At end)
[2021-04-14 16:53:20,165] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]    input_df= (20088, 12) 
[2021-04-14 16:53:20,165] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]    current_mf=(0, 0) 
[2021-04-14 16:53:20,165] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]    PPR: 
[2021-04-14 16:53:20,165] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]       UNPROCESSED = <class 'pandas.core.frame.DataFrame'> ((20088, 12))
[2021-04-14 16:53:20,165] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]       TRAIN = <class 'dataiku.doctor.multiframe.MultiFrame'> ((20088, 222))
[2021-04-14 16:53:20,165] [482260/MainThread] [DEBUG] [dku.ml.preprocessing]       target = <class 'pandas.core.series.Series'> ((20088,))
[2021-04-14 16:53:20,165] [482260/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Preprocessing test set
[2021-04-14 16:53:20,166] [482260/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Hyperparameter searching
[2021-04-14 16:53:20,274] [482260/MainThread] [DEBUG] [dku.ml.preprocessing] Deleting 0 rows
[2021-04-14 16:53:20,274] [482260/MainThread] [INFO] [dataiku.doctor.multiframe] MultiFrame, dropping rows: []
[2021-04-14 16:53:20,288] [482260/MainThread] [INFO] [dataiku.doctor.prediction.common] prepare multiframe shape=(79912,222) tn=17740464 nnz=820660 fill_ratio=0.05
[2021-04-14 16:53:20,288] [482260/MainThread] [INFO] [dataiku.doctor.prediction.common] too small, using array
[2021-04-14 16:53:20,396] [482260/MainThread] [INFO] [dataiku.doctor.prediction.common] Create CLF from params: {u'computeLearningCurves': False, u'ridge_grid': {u'alpha': {u'values': [0.1, 1.0, 3.0], u'gridMode': u'EXPLICIT', u'randomMode': u'RANGE', u'limit': {u'min': 1e-23}, u'range': {u'scaling': u'LOGARITHMIC', u'max': 3.0, u'nbValues': 3, u'min': 0.1}}, u'alphaMode': u'MANUAL', u'enabled': True}, u'algorithm': u'RIDGE_REGRESSION', u'grid_search_params': {u'nIter': 0, u'nJobs': 4, u'nContainers': 4, u'distributed': False, u'nIterRandom': 3, u'bayesianOptimizer': u'SCIKIT_OPTIMIZE', u'strategy': u'GRID', u'randomized': True, u'shuffleIterations': 1, u'seed': 0, u'mode': u'KFOLD', u'timeout': 0, u'splitRatio': 0.8, u'nFolds': 5, u'stratified': True}, u'autoOptimizeThreshold': True, u'gridLength': 3, u'metrics': {u'customEvaluationMetricNeedsProba': False, u'evaluationMetric': u'R2', u'liftPoint': 0.4, u'costMatrixWeights': {u'fnGain': 0.0, u'tpGain': 1.0, u'tnGain': 0.0, u'fpGain': -0.3}, u'customEvaluationMetricGIB': True}, u'forcedClassifierThreshold': 0.0, u'skipExpensiveReports': False, u'max_ensemble_nodes_serialized': 0, u'pluginAlgoCustomGridSearch': False} for algorithm RIDGE_REGRESSION
[2021-04-14 16:53:20,396] [482260/MainThread] [INFO] [dataiku.doctor.prediction.common] Using K-Fold CV with k=5
[2021-04-14 16:53:20,402] [482260/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluation_monitor] No distributed container configuration is available to run this search
[2021-04-14 16:53:20,407] [482260/MainThread] [INFO] [dataiku.doctor.crossval.search_runner] Fitting 5 folds for each of 3 candidates, totalling 15 fits
[2021-04-14 16:53:20,407] [482260/MainThread] [INFO] [dataiku.doctor.crossval.search_runner] Execute hyperparameter search locally on 4 threads
[2021-04-14 16:53:20,408] [482260/MainThread] [INFO] [dataiku.doctor.crossval.strategies.grid_search_strategy] Running GridSearchStrategy for hyperparameters space: <dataiku.doctor.prediction.common.HyperparametersSpace object at 0x7fa807fb5f90>
[2021-04-14 16:53:20,408] [482260/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 0: {'alpha': 0.1}
[2021-04-14 16:53:20,408] [482260/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.local_worker] Starting worker: local-0
[2021-04-14 16:53:20,408] [482260/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.local_worker] Started worker: local-0
[2021-04-14 16:53:20,408] [482260/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 1: {'alpha': 0.1}
[2021-04-14 16:53:20,408] [482260/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 16:53:20,408] [482260/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.local_worker] Starting worker: local-1
[2021-04-14 16:53:20,409] [482260/Thread-2:local-0] [INFO] [dataiku.doctor.crossval.search_context] Fit s=0: alpha=0.1 .......................................................
[2021-04-14 16:53:20,409] [482260/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 2: {'alpha': 0.1}
[2021-04-14 16:53:20,409] [482260/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.local_worker] Started worker: local-1
[2021-04-14 16:53:20,409] [482260/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.local_worker] Starting worker: local-2
[2021-04-14 16:53:20,409] [482260/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 16:53:20,409] [482260/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.local_worker] Started worker: local-2
[2021-04-14 16:53:20,410] [482260/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 3: {'alpha': 0.1}
[2021-04-14 16:53:20,410] [482260/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Fit s=1: alpha=0.1 .......................................................
[2021-04-14 16:53:20,410] [482260/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 16:53:20,410] [482260/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.local_worker] Starting worker: local-3
[2021-04-14 16:53:20,411] [482260/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 4: {'alpha': 0.1}
[2021-04-14 16:53:20,411] [482260/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Fit s=2: alpha=0.1 .......................................................
[2021-04-14 16:53:20,411] [482260/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.local_worker] Started worker: local-3
[2021-04-14 16:53:20,412] [482260/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 16:53:20,413] [482260/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Fit s=3: alpha=0.1 .......................................................
[2021-04-14 16:53:21,034] [482260/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Done s=3: alpha=0.1 (ft=0.0s st=0.0s sc=0.844925669538, sg=1)
[2021-04-14 16:53:21,051] [482260/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 16:53:21,052] [482260/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 16:53:21,053] [482260/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Fit s=4: alpha=0.1 .......................................................
[2021-04-14 16:53:21,053] [482260/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 0: {'alpha': 1.0}
[2021-04-14 16:53:21,179] [482260/Thread-2:local-0] [INFO] [dataiku.doctor.crossval.search_context] Done s=0: alpha=0.1 (ft=0.0s st=0.0s sc=0.84577330003, sg=1)
[2021-04-14 16:53:21,197] [482260/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 16:53:21,198] [482260/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Done s=2: alpha=0.1 (ft=0.0s st=0.0s sc=0.849061981051, sg=1)
[2021-04-14 16:53:21,219] [482260/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 16:53:21,219] [482260/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 16:53:21,220] [482260/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Fit s=0: alpha=1.0 .......................................................
[2021-04-14 16:53:21,221] [482260/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Done s=1: alpha=0.1 (ft=0.0s st=0.0s sc=0.85385316636, sg=1)
[2021-04-14 16:53:21,238] [482260/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 16:53:21,239] [482260/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 1: {'alpha': 1.0}
[2021-04-14 16:53:21,239] [482260/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 2: {'alpha': 1.0}
[2021-04-14 16:53:21,239] [482260/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 3: {'alpha': 1.0}
[2021-04-14 16:53:21,239] [482260/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 16:53:21,240] [482260/Thread-2:local-0] [INFO] [dataiku.doctor.crossval.search_context] Fit s=1: alpha=1.0 .......................................................
[2021-04-14 16:53:21,241] [482260/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 16:53:21,241] [482260/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Fit s=2: alpha=1.0 .......................................................
[2021-04-14 16:53:21,719] [482260/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Done s=4: alpha=0.1 (ft=0.0s st=0.0s sc=0.845759984587, sg=1)
[2021-04-14 16:53:21,733] [482260/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 16:53:21,734] [482260/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 16:53:21,734] [482260/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Fit s=3: alpha=1.0 .......................................................
[2021-04-14 16:53:21,934] [482260/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Done s=2: alpha=1.0 (ft=0.0s st=0.0s sc=0.673991280839, sg=1)
[2021-04-14 16:53:21,947] [482260/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 16:53:21,948] [482260/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Done s=0: alpha=1.0 (ft=0.0s st=0.0s sc=0.6696726809, sg=1)
[2021-04-14 16:53:21,975] [482260/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 16:53:21,998] [482260/Thread-2:local-0] [INFO] [dataiku.doctor.crossval.search_context] Done s=1: alpha=1.0 (ft=0.0s st=0.0s sc=0.676000330888, sg=1)
[2021-04-14 16:53:22,014] [482260/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 16:53:22,163] [482260/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Done s=3: alpha=1.0 (ft=0.0s st=0.0s sc=0.669560206914, sg=1)
[2021-04-14 16:53:22,176] [482260/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 16:53:22,294] [482260/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 4: {'alpha': 1.0}
[2021-04-14 16:53:22,294] [482260/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 16:53:22,295] [482260/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Fit s=4: alpha=1.0 .......................................................
[2021-04-14 16:53:22,462] [482260/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 0: {'alpha': 3.0}
[2021-04-14 16:53:22,462] [482260/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 1: {'alpha': 3.0}
[2021-04-14 16:53:22,462] [482260/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 2: {'alpha': 3.0}
[2021-04-14 16:53:22,462] [482260/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 3: {'alpha': 3.0}
[2021-04-14 16:53:22,462] [482260/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 16:53:22,463] [482260/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Fit s=0: alpha=3.0 .......................................................
[2021-04-14 16:53:22,467] [482260/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 16:53:22,468] [482260/Thread-2:local-0] [INFO] [dataiku.doctor.crossval.search_context] Fit s=1: alpha=3.0 .......................................................
[2021-04-14 16:53:22,469] [482260/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 16:53:22,469] [482260/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Fit s=2: alpha=3.0 .......................................................
[2021-04-14 16:53:22,625] [482260/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Done s=4: alpha=1.0 (ft=0.0s st=0.0s sc=0.671061168427, sg=1)
[2021-04-14 16:53:22,638] [482260/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 16:53:22,639] [482260/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 16:53:22,641] [482260/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Fit s=3: alpha=3.0 .......................................................
[2021-04-14 16:53:22,643] [482260/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 4: {'alpha': 3.0}
[2021-04-14 16:53:23,212] [482260/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Done s=3: alpha=3.0 (ft=0.0s st=0.0s sc=0.429207028389, sg=1)
[2021-04-14 16:53:23,227] [482260/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 16:53:23,228] [482260/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Done s=2: alpha=3.0 (ft=0.0s st=0.0s sc=0.432687829485, sg=1)
[2021-04-14 16:53:23,230] [482260/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Done s=0: alpha=3.0 (ft=0.0s st=0.0s sc=0.429156825352, sg=1)
[2021-04-14 16:53:23,243] [482260/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 16:53:23,244] [482260/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 16:53:23,258] [482260/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 16:53:23,259] [482260/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Fit s=4: alpha=3.0 .......................................................
[2021-04-14 16:53:23,260] [482260/Thread-2:local-0] [INFO] [dataiku.doctor.crossval.search_context] Done s=1: alpha=3.0 (ft=0.0s st=0.0s sc=0.432532469409, sg=1)
[2021-04-14 16:53:23,272] [482260/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 16:53:23,556] [482260/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Done s=4: alpha=3.0 (ft=0.0s st=0.0s sc=0.430011649371, sg=1)
[2021-04-14 16:53:23,569] [482260/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 16:53:24,411] [482260/Thread-1] [INFO] [dataiku.doctor.crossval.search_runner] Completed search for hyperparameters
[2021-04-14 16:53:24,411] [482260/Thread-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Scheduler has been soft interrupted
[2021-04-14 16:53:24,411] [482260/MainThread] [INFO] [dataiku.doctor.distributed.work_scheduler] Scheduler has been hard interrupted (shutdown)
[2021-04-14 16:53:24,412] [482260/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.local_worker] Stopping worker: local-1
[2021-04-14 16:53:24,412] [482260/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.local_worker] Stopping worker: local-2
[2021-04-14 16:53:24,412] [482260/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.local_worker] Stopped worker: local-1
[2021-04-14 16:53:24,412] [482260/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.local_worker] Stopping worker: local-3
[2021-04-14 16:53:24,412] [482260/Thread-2:local-0] [ERROR] [dataiku.doctor.distributed.work_scheduler] All workers are dead, interrupt the scheduler
[2021-04-14 16:53:24,412] [482260/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.local_worker] Stopped worker: local-2
[2021-04-14 16:53:24,412] [482260/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.local_worker] Stopped worker: local-3
[2021-04-14 16:53:24,412] [482260/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.local_worker] Stopping worker: local-0
[2021-04-14 16:53:24,413] [482260/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.local_worker] Stopped worker: local-0
[2021-04-14 16:53:24,413] [482260/MainThread] [INFO] [dataiku.doctor.crossval.search_runner] Hyperparameter search done, best_parameters being : {'alpha': 0.1}
[2021-04-14 16:53:24,413] [482260/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Hyperparameter searching
[2021-04-14 16:53:24,413] [482260/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Fitting model
[2021-04-14 16:53:24,445] [482260/MainThread] [INFO] [dataiku.doctor.prediction.common] Fitting model:
[2021-04-14 16:53:24,446] [482260/MainThread] [INFO] [dataiku.doctor.prediction.common]   Model is: Ridge(alpha=0.1, copy_X=True, fit_intercept=True, max_iter=None,
   normalize=True, random_state=None, solver='auto', tol=0.001)
[2021-04-14 16:53:24,446] [482260/MainThread] [INFO] [dataiku.doctor.prediction.common]   train_X class: <type 'numpy.ndarray'>
[2021-04-14 16:53:24,446] [482260/MainThread] [INFO] [dataiku.doctor.prediction.common]   train_X shape: (79912, 222)
[2021-04-14 16:53:24,446] [482260/MainThread] [INFO] [dataiku.doctor.prediction.common]   train_y shape: (79912,)
[2021-04-14 16:53:24,446] [482260/MainThread] [INFO] [dataiku.doctor.prediction.common]   calibration enabled: a sub-sample of the train data has been saved for calibration
[2021-04-14 16:53:24,688] [482260/MainThread] [INFO] [dataiku.doctor.prediction.regression_fit] Output params are {'resolved': {'ridge': {'alpha': 0.1}, 'skipExpensiveReports': False, 'algorithm': u'RIDGE_REGRESSION'}, 'other': {}}
[2021-04-14 16:53:24,688] [482260/MainThread] [INFO] [dataiku.doctor.prediction.scoring_base] No feature importance in classifier
[2021-04-14 16:53:24,689] [482260/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Fitting model
[2021-04-14 16:53:24,689] [482260/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Saving model
[2021-04-14 16:53:24,690] [482260/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Saving model
[2021-04-14 16:53:24,690] [482260/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Scoring model
[2021-04-14 16:53:24,750] [482260/MainThread] [INFO] [dataiku.doctor.prediction.regression_scoring] Intrinsic scoring
[2021-04-14 16:53:24,750] [482260/MainThread] [INFO] [dataiku.doctor.prediction.regression_scoring] Extracting rescalers
[2021-04-14 16:53:24,750] [482260/MainThread] [INFO] [dataiku.doctor.prediction.scoring_base] Computing feature importance
[2021-04-14 16:53:24,750] [482260/MainThread] [INFO] [dataiku.doctor.prediction.scoring_base] No feature importance in classifier
[2021-04-14 16:53:24,750] [482260/MainThread] [INFO] [dataiku.doctor.prediction.regression_scoring] Computing regression coeffs
[2021-04-14 16:53:24,892] [482260/MainThread] [INFO] [dataiku.doctor.prediction.scoring_base] Coefs: [ 2.51468227e-01  7.77236380e-04  4.03962354e-04  1.64199894e-03
 -1.08709623e-04  2.21274246e-04 -2.14148734e-03 -2.34767962e-03
 -3.82966274e-03  9.98070240e-02  9.75567014e-03  3.31335906e-02
 -2.11422257e-02 -5.18877387e-02 -3.49659251e-03  7.58471781e-02
  1.88267621e-02  8.09517207e-03  1.07772388e-01 -1.39270152e-03
 -3.57200870e-02  8.02217348e-03  1.66063943e-02  1.25685898e-01
  3.88042401e-02  1.02873595e-01 -1.64244410e-02  2.74622539e-02
 -1.08666499e-02 -3.45813610e-02  9.45307225e-02  3.84363336e-02
  4.07360609e-02 -7.42249698e-02 -3.19906176e-02  3.10738280e-02
 -5.55059820e-02 -1.26285997e-01  3.40500570e-02 -8.33409399e-02
  6.34217095e-03 -6.44124456e-02 -6.80246641e-02 -1.32911237e-02
 -1.43996297e-02 -2.19741175e-02 -7.09612515e-02 -8.18414036e-02
 -5.25380754e-02 -4.11381462e-02 -1.83724241e-02  5.17067456e-02
 -5.08883004e-02 -3.42208079e-02 -7.03493232e-02 -8.68218962e-03
 -3.41980990e-02 -8.35231289e-02 -5.81115231e-03  1.31572863e-02
  1.90821627e-05 -7.14890835e-02 -1.76929128e-02 -6.18459976e-02
  2.93512095e-02 -4.81391755e-02 -6.29969730e-02 -2.53321339e-02
 -1.22495673e-01 -3.71033064e-02 -4.39063444e-02 -2.98396516e-03
 -4.32685204e-02 -4.08506624e-02 -2.35085271e-02 -3.80761615e-02
 -1.00103476e-02 -8.49355662e-02  7.68190347e-02  1.91085776e-02
 -5.51398058e-02 -2.60883629e-02 -1.26046814e-02 -1.10980232e-02
 -5.09033918e-02 -1.12712247e-02  9.65428784e-03 -1.18080499e-01
 -9.06477766e-02 -3.16010689e-02 -8.76639958e-02 -3.94115214e-02
 -7.21416967e-02 -5.47438453e-03 -1.17706378e-01 -7.20235677e-03
 -2.40185171e-02 -4.26247393e-02 -3.88573679e-02  1.09229599e-01
 -3.01100269e-02 -9.12895375e-03  1.76899650e-02 -4.33922914e-02
 -1.50969693e-02  5.36260679e-02 -8.30613708e-02 -9.53318734e-02
 -9.21465170e-03  0.00000000e+00 -3.24730148e-02 -4.22604505e-03
 -7.20309943e-04 -2.18920001e-03 -1.14234605e-02  3.29788891e-05
  7.30681242e-04  0.00000000e+00  4.75764062e-03  4.32147178e-03
 -2.86535557e-03 -4.65041817e-03 -9.32615093e-03 -6.87646643e-04
 -4.95681849e-03  2.28017685e-03 -3.11651728e-03 -2.32463315e-03
  8.19631864e-04  4.67293058e-03 -1.38053973e-02 -4.01895304e-03
  4.25728159e-03 -7.23952727e-04  2.87586263e-03  5.67337001e-03
 -1.88806788e-03  1.53550678e-03  3.64872846e-03  1.48140069e-03
  2.62924135e-03  4.21307126e-03 -1.07848051e-03  1.24906244e-03
  3.15273824e-03  1.35244381e-03  5.19588278e-03  1.53125706e-03
  4.70171529e-03 -3.25469878e-03 -4.64736375e-02  1.84061106e-03
 -3.74564095e-03  5.93114075e-03  5.15937285e-03  5.92609436e-04
 -2.22304078e-03 -3.67764998e-03 -2.30281466e-03  1.51615530e-03
  8.14725658e-04 -3.87742544e-03  2.28665191e-03  5.89442819e-03
  1.71283593e-03 -3.30652982e-03  3.64447680e-03  2.81240696e-03
  7.11149514e-03 -1.26385258e-03  5.92143852e-03  2.95752819e-03
  6.44867090e-03 -5.70671532e-04  7.63780336e-03 -2.19765582e-02
  6.73404154e-03 -2.20520039e-03  1.97188216e-03  2.15762443e-03
  4.33604398e-03  4.76350911e-04  5.29677343e-03  2.06638807e-03
 -2.43695182e-03  9.07737075e-03  3.14607722e-03  9.31668615e-04
  7.01738471e-03  6.30329525e-03 -2.69053938e-02  8.50156082e-03
  2.56467361e-03  2.85272154e-03 -2.22448582e-02 -3.74520330e-03
  5.99708827e-03  5.70185935e-03 -1.77872151e-03  1.78373898e-03
 -1.84764111e-03  6.08697599e-03  1.92738100e-04  2.07217650e-03
  4.05697434e-03 -5.78676706e-05  4.03721741e-03 -7.01036674e-04
  5.25232418e-03  5.83651353e-03  6.75942194e-04  4.43040596e-03
  1.97149638e-03  3.68404999e-03  4.02620778e-03  2.83650485e-03
  6.46883910e-03  9.93457037e-03  8.25600108e-03  0.00000000e+00
  1.76753210e-03 -5.56283182e-03  0.00000000e+00]
[2021-04-14 16:53:24,892] [482260/MainThread] [INFO] [dataiku.doctor.prediction.scoring_base] Sample variance: 0.0006884053809200157
[2021-04-14 16:53:24,985] [482260/MainThread] [INFO] [dataiku.doctor.prediction.scoring_base] Singular variance matrix
[2021-04-14 16:53:25,121] [482260/MainThread] [INFO] [dataiku.doctor.prediction.common] prepare multiframe shape=(20088,222) tn=4459536 nnz=206420 fill_ratio=0.05
[2021-04-14 16:53:25,121] [482260/MainThread] [INFO] [dataiku.doctor.prediction.common] too small, using array
[2021-04-14 16:53:25,146] [482260/MainThread] [INFO] [dataiku.doctor.prediction.regression_scoring] Creating predictions on test set
[2021-04-14 16:53:25,153] [482260/MainThread] [INFO] [dataiku.doctor.prediction.regression_scoring] Computing regression performance on [0.2486259  0.2486259  0.2486259  ... 0.22897263 0.23421826 0.24113754]
[2021-04-14 16:53:25,520] [482260/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Scoring model
[2021-04-14 16:53:25,520] [482260/MainThread] [INFO] [root] Reading with dtypes: None
[2021-04-14 16:53:25,520] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_month: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 16:53:25,520] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_woy: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 16:53:25,520] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_dow: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 16:53:25,520] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_hour: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 16:53:25,520] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for UniqueCarrier: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 16:53:25,520] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for FlightNum: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 16:53:25,520] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for AirTime: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 16:53:25,520] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Late: str (schema_type=boolean feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 16:53:25,521] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Origin: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 16:53:25,521] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Dest: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 16:53:25,521] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Distance: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 16:53:25,521] [482260/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Late_avg: <type 'numpy.float64'> (schema_type=double feature_type=NUMERIC feature_role=TARGET)
[2021-04-14 16:53:25,521] [482260/MainThread] [INFO] [root] Reading with FIXED dtypes: {u'dep_hour': <type 'numpy.float64'>, u'Late': 'str', u'UniqueCarrier': 'str', u'FlightNum': <type 'numpy.float64'>, u'Late_avg': <type 'numpy.float64'>, u'dep_woy': <type 'numpy.float64'>, u'Distance': <type 'numpy.float64'>, u'Dest': 'str', u'Origin': 'str', u'dep_month': <type 'numpy.float64'>, u'dep_dow': <type 'numpy.float64'>, u'AirTime': <type 'numpy.float64'>}
[2021-04-14 16:53:25,549] [482260/MainThread] [INFO] [root] Loaded table
[2021-04-14 16:53:25,551] [482260/MainThread] [INFO] [dataiku.doctor.prediction.background_rows_handler] Building background rows
[2021-04-14 16:53:25,552] [482260/MainThread] [INFO] [dataiku.doctor.prediction.background_rows_handler] Background estimated bias <= 0.000771176674222
[2021-04-14 16:53:25,555] [482260/MainThread] [INFO] [dataiku.doctor.prediction.background_rows_handler] Built background rows with shape=(1000, 12)
[2021-04-14 16:53:25,555] [482260/MainThread] [INFO] [dataiku.doctor.prediction.background_rows_handler] Saving background rows with shape=(1000, 12)
[2021-04-14 16:53:25,562] [482260/MainThread] [INFO] [dataiku.doctor.utils]  Coercion done
[2021-04-14 16:53:25,563] [482260/MainThread] [INFO] [dataiku.doctor.prediction.histogram_handler] Computing histograms
[2021-04-14 16:53:25,728] [482260/MainThread] [INFO] [dataiku.doctor.prediction.histogram_handler] Saving histograms
[2021-04-14 16:53:25,749] [482260/MainThread] [INFO] [dataiku.doctor.prediction.column_importance_handler] Computing column importance
/data/home/dkuacademy/Applications/dataiku-dss-9.0.0/python/dataiku/doctor/prediction/column_importance_handler.py:132: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  clf.fit(preprocessed_array[idx], target[idx])
[2021-04-14 16:53:26,272] [482260/MainThread] [INFO] [dataiku.doctor.prediction.column_importance_handler] Saving column importance
[2021-04-14 16:53:26,289] [482260/MainThread] [INFO] [dataiku.base.socket_block_link] Client closed
[2021/04/14-16:53:26.289] [MRT-280194] [INFO] [dku.block.link.interaction]  - Check result for nullity exceptionIfNull=true result=not null
[2021/04/14-16:53:26.289] [MRT-280194] [INFO] [dku.analysis.prediction]  - Training returned ok
[2021/04/14-16:53:26.289] [MRT-280194] [INFO] [dku.ml.versioning]  - Dumping version info {"trainedWithDSSVersion":"9.0.0","trainedWithDSSConfVersion":"9000","trainedWithDSSBackend":"PY_MEMORY","backendCompatibilityVersion":18} in /data/home/dkuacademy/Library/dss-9.0/design/analysis-data/QS_DATAENG2/oSylVxXB/0IfwLWGQ/sessions/s4/pp1/m1
[2021/04/14-16:53:26.290] [MRT-280194] [INFO] [dku.analysis.ml]  - preTrain={
  "algorithm": "RIDGE_REGRESSION",
  "ridge_grid": {
    "alphaMode": "MANUAL",
    "alpha": {
      "gridMode": "EXPLICIT",
      "randomMode": "RANGE",
      "limit": {
        "min": 1.0E-23
      },
      "values": [
        0.1,
        1.0,
        3.0
      ],
      "range": {
        "min": 0.1,
        "max": 3.0,
        "nbValues": 3,
        "scaling": "LOGARITHMIC"
      }
    },
    "enabled": true
  },
  "max_ensemble_nodes_serialized": 0,
  "metrics": {
    "evaluationMetric": "R2",
    "customEvaluationMetricGIB": true,
    "customEvaluationMetricNeedsProba": false,
    "costMatrixWeights": {
      "tpGain": 1.0,
      "tnGain": 0.0,
      "fpGain": -0.3,
      "fnGain": 0.0
    },
    "liftPoint": 0.4
  },
  "autoOptimizeThreshold": true,
  "forcedClassifierThreshold": 0.0,
  "gridLength": 3,
  "grid_search_params": {
    "mode": "KFOLD",
    "splitRatio": 0.8,
    "shuffleIterations": 1,
    "nFolds": 5,
    "stratified": true,
    "strategy": "GRID",
    "bayesianOptimizer": "SCIKIT_OPTIMIZE",
    "randomized": true,
    "seed": 0,
    "nIter": 0,
    "nIterRandom": 3,
    "timeout": 0,
    "nJobs": 4,
    "distributed": false,
    "nContainers": 4
  },
  "pluginAlgoCustomGridSearch": false,
  "computeLearningCurves": false,
  "skipExpensiveReports": false
}
[2021/04/14-16:53:26.290] [MRT-280194] [INFO] [dku.analysis.ml]  - amp={
  "resolved": {
    "algorithm": "RIDGE_REGRESSION",
    "ridge": {
      "alpha": 0.1
    },
    "skipExpensiveReports": false
  },
  "other": {}
}
[2021/04/14-16:53:26.372] [KNL-python-single-command-kernel-monitor-280198] [INFO] [dku.kernels]  - Process done with code 0
[2021/04/14-16:53:26.373] [KNL-python-single-command-kernel-monitor-280198] [INFO] [dip.tickets]  - Destroying API ticket for analysis-ml-QS_DATAENG2-1d2hoIW on behalf of tara
[2021/04/14-16:53:26.373] [KNL-python-single-command-kernel-monitor-280198] [WARN] [dku.resource]  - stat file for pid 482260 does not exist. Process died?
[2021/04/14-16:53:26.374] [KNL-python-single-command-kernel-monitor-280198] [INFO] [dku.resourceusage]  - Reporting completion of CRU:{"context":{"type":"ANALYSIS_ML_TRAIN","authIdentifier":"tara","projectKey":"QS_DATAENG2","analysisId":"oSylVxXB","mlTaskId":"0IfwLWGQ","sessionId":"s4"},"type":"LOCAL_PROCESS","id":"WgxeZ7Ve8u3KpLMh","startTime":1618411986434,"localProcess":{"pid":482260,"commandName":"/data/home/dkuacademy/Library/dss-9.0/design/bin/python","cpuUserTimeMS":4550,"cpuSystemTimeMS":4000,"cpuChildrenUserTimeMS":0,"cpuChildrenSystemTimeMS":0,"cpuTotalMS":8550,"cpuCurrent":1.390053924505692,"vmSizeMB":3645,"vmRSSMB":1289,"vmHWMMB":1290,"vmRSSAnonMB":1269,"vmDataMB":2984,"vmSizePeakMB":3649,"vmRSSPeakMB":1289,"vmRSSTotalMBS":7295,"majorFaults":170,"childrenMajorFaults":2}}
[2021/04/14-16:53:26.484] [MRT-280194] [INFO] [dku.block.link]  - Closed socket
[2021/04/14-16:53:26.484] [MRT-280194] [INFO] [dku.block.link]  - Closed socket
[2021/04/14-16:53:26.484] [MRT-280194] [INFO] [dku.block.link]  - Closed serverSocket
[2021/04/14-16:53:26.484] [FT-TrainWorkThread-4VpvYhxi-280193] [INFO] [dku.analysis.ml.python] T-0IfwLWGQ - Processing thread joined ...
[2021/04/14-16:53:26.484] [FT-TrainWorkThread-4VpvYhxi-280193] [INFO] [dku.analysis.ml.python] T-0IfwLWGQ - Joining processing thread ...
[2021/04/14-16:53:26.485] [FT-TrainWorkThread-4VpvYhxi-280193] [INFO] [dku.analysis.ml.python] T-0IfwLWGQ - Processing thread joined ...
[2021/04/14-16:53:26.485] [FT-TrainWorkThread-4VpvYhxi-280193] [INFO] [dku.analysis.prediction] T-0IfwLWGQ - Train done
[2021/04/14-16:53:26.485] [FT-TrainWorkThread-4VpvYhxi-280193] [INFO] [dku.analysis.prediction] T-0IfwLWGQ - Train done
[2021/04/14-16:53:26.487] [FT-TrainWorkThread-4VpvYhxi-280193] [INFO] [dku.analysis.trainingdetails] T-0IfwLWGQ - Publishing mltask-train-done reflected event
