[2021/04/14-19:03:30.330] [FT-TrainWorkThread-i4H0sQww-284009] [INFO] [dku.analysis.prediction]  - ******************************************
[2021/04/14-19:03:30.330] [FT-TrainWorkThread-i4H0sQww-284009] [INFO] [dku.analysis.prediction]  - ** Start train session s7
[2021/04/14-19:03:30.330] [FT-TrainWorkThread-i4H0sQww-284009] [INFO] [dku.analysis.prediction]  - ******************************************
[2021/04/14-19:03:30.331] [FT-TrainWorkThread-i4H0sQww-284009] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 1] Search for split: p=type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337 i=bf10335945b1f843a4ae9838d57674b3-0
[2021/04/14-19:03:30.331] [FT-TrainWorkThread-i4H0sQww-284009] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 1] Checking if splits are up to date. Policy: type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337, instance id: bf10335945b1f843a4ae9838d57674b3-0
[2021/04/14-19:03:30.332] [FT-TrainWorkThread-i4H0sQww-284009] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 2] Search for split: p=type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337 i=bf10335945b1f843a4ae9838d57674b3-0
[2021/04/14-19:03:30.332] [FT-TrainWorkThread-i4H0sQww-284009] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 2] Search for split: p=type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337 i=bf10335945b1f843a4ae9838d57674b3-0
[2021/04/14-19:03:30.333] [FT-TrainWorkThread-i4H0sQww-284009] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 3] Checking if splits are up to date. Policy: type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337, instance id: bf10335945b1f843a4ae9838d57674b3-0
[2021/04/14-19:03:30.333] [FT-TrainWorkThread-i4H0sQww-284009] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 3] Search for split: p=type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337 i=bf10335945b1f843a4ae9838d57674b3-0
[2021/04/14-19:03:30.334] [FT-TrainWorkThread-i4H0sQww-284009] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 4] Search for split: p=type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337 i=bf10335945b1f843a4ae9838d57674b3-0
[2021/04/14-19:03:30.334] [FT-TrainWorkThread-i4H0sQww-284009] [INFO] [dku.analysis.ml.python] T-0IfwLWGQ - Joining processing thread ...
[2021/04/14-19:03:30.336] [MRT-284010] [INFO] [dku.analysis.ml.python]  - TrainAdditionalThread done
[2021/04/14-19:03:30.336] [MRT-284011] [INFO] [dku.analysis.ml.python]  - Running a preprocessing set: pp1 in /data/home/dkuacademy/Library/dss-9.0/design/analysis-data/QS_DATAENG2/oSylVxXB/0IfwLWGQ/sessions/s7/pp1
[2021/04/14-19:03:30.336] [FT-TrainWorkThread-i4H0sQww-284009] [INFO] [dku.analysis.ml.python] T-0IfwLWGQ - Processing thread joined ...
[2021/04/14-19:03:30.336] [FT-TrainWorkThread-i4H0sQww-284009] [INFO] [dku.analysis.ml.python] T-0IfwLWGQ - Joining processing thread ...
[2021/04/14-19:03:30.336] [MRT-284011] [INFO] [dku.block.link]  - Started a socket on port 46514
[2021/04/14-19:03:30.336] [MRT-284011] [INFO] [dku.ml.kernel]  - Writing output of python-single-command-kernel to /data/home/dkuacademy/Library/dss-9.0/design/analysis-data/QS_DATAENG2/oSylVxXB/0IfwLWGQ/sessions/s7/pp1/train.log
[2021/04/14-19:03:30.337] [MRT-284011] [INFO] [dku.code.envs.resolution]  - Executing Python activity in env: py_36_tara
[2021/04/14-19:03:30.337] [MRT-284011] [WARN] [dku.code.projectLibs]  - External libraries file not found: /data/home/dkuacademy/Library/dss-9.0/design/config/projects/QS_DATAENG2/lib/external-libraries.json
[2021/04/14-19:03:30.337] [MRT-284011] [INFO] [dku.code.projectLibs]  - EXTERNAL LIBS FROM QS_DATAENG2 is {"gitReferences":{},"pythonPath":["python"],"rsrcPath":["R"],"importLibrariesFromProjects":[]}
[2021/04/14-19:03:30.337] [MRT-284011] [INFO] [dku.code.projectLibs]  - chunkFolder is /data/home/dkuacademy/Library/dss-9.0/design/config/projects/QS_DATAENG2/lib/R
[2021/04/14-19:03:30.337] [MRT-284011] [INFO] [dku.python.single_command.kernel]  - Starting Python process for kernel  python-single-command-kernel
[2021/04/14-19:03:30.337] [MRT-284011] [INFO] [dip.tickets]  - Creating API ticket for analysis-ml-QS_DATAENG2-vdBizv6 on behalf of tara id=analysis-ml-QS_DATAENG2-vdBizv6_sfdVSqeI55tk
[2021/04/14-19:03:30.337] [MRT-284011] [INFO] [dku.security.process]  - Starting process (regular)
[2021/04/14-19:03:30.339] [MRT-284011] [INFO] [dku.security.process]  - Process started with pid=7842
[2021/04/14-19:03:30.340] [MRT-284011] [INFO] [dku.processes.cgroups]  - Will use cgroups []
[2021/04/14-19:03:30.340] [MRT-284011] [INFO] [dku.processes.cgroups]  - Applying rules to used cgroups: []
[2021/04/14-19:03:30.341] [KNL-python-single-command-kernel-monitor-284014] [INFO] [dku.resourceusage]  - Reporting start of CRU:{"context":{"type":"ANALYSIS_ML_TRAIN","authIdentifier":"tara","projectKey":"QS_DATAENG2","analysisId":"oSylVxXB","mlTaskId":"0IfwLWGQ","sessionId":"s7"},"type":"LOCAL_PROCESS","id":"6YY9eDNH1XA1Qwng","startTime":1618419810341,"localProcess":{"cpuCurrent":0.0}}
[2021/04/14-19:03:30.347] [process-resource-monitor-7842-284015] [DEBUG] [dku.resource]  - Process stats for pid 7842: {"pid":7842,"commandName":"/data/home/dkuacademy/Library/dss-9.0/design/code-envs/python/py_36_tara/bin/python","cpuUserTimeMS":0,"cpuSystemTimeMS":0,"cpuChildrenUserTimeMS":0,"cpuChildrenSystemTimeMS":0,"cpuTotalMS":0,"cpuCurrent":0.0,"vmSizeMB":121,"vmRSSMB":4,"vmHWMMB":4,"vmRSSAnonMB":2,"vmDataMB":2,"vmSizePeakMB":121,"vmRSSPeakMB":4,"vmRSSTotalMBS":0,"majorFaults":0,"childrenMajorFaults":0}
Installing debugging signal handler
[2021-04-14 19:03:31,138] [7842/MainThread] [INFO] [dataiku.base.socket_block_link] Connecting to localhost (127.0.0.1) at port 46514
[2021-04-14 19:03:31,138] [7842/MainThread] [INFO] [dataiku.base.socket_block_link] Connected to localhost (127.0.0.1) at port 46514
[2021/04/14-19:03:31.139] [MRT-284011] [INFO] [dku.link.secret_protected]  - Connected to kernel
[2021/04/14-19:03:31.142] [MRT-284011] [INFO] [dku.block.link.interaction]  - Execute link command respClazz=true respTypeToken=false respIsString=false is=false asyncInputStream=false os=false
[2021-04-14 19:03:31,409] [7842/MainThread] [INFO] [dataiku.doctor.utils.dku_pickle] Setting cloudpickle as the pickling tool
/data/home/dkuacademy/Applications/dataiku-dss-9.0.0/python/dataiku/doctor/dkuapi.py:16: DeprecationWarning: inspect.getargspec() is deprecated since Python 3.0, use inspect.signature() or inspect.getfullargspec()
  argspec = inspect.getargspec(api)
[2021-04-14 19:03:31,431] [7842/MainThread] [INFO] [root] Running analysis command: train_prediction_models_nosave
[2021-04-14 19:03:31,432] [7842/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: DatasetSanityCheckDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_DATASET_SANITY_CHECKS
[2021-04-14 19:03:31,432] [7842/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: ClassifierAccuracyCheckDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_MODEL_CHECK
[2021-04-14 19:03:31,432] [7842/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: RegressionR2CheckDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_MODEL_CHECK
[2021-04-14 19:03:31,432] [7842/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: LeakageDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_LEAKAGE_DETECTION
[2021-04-14 19:03:31,432] [7842/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: TreeOverfitDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_TRAINING_OVERFIT
[2021-04-14 19:03:31,432] [7842/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: MLAssertionsDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_ML_ASSERTIONS
[2021-04-14 19:03:31,432] [7842/MainThread] [INFO] [dataiku.doctor.commands] PPS is {'target_remapping': [], 'skipPreprocessing': False, 'per_feature': {'FlightNum': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Origin': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 100, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'NONE', 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Text'}, 'autoReason': 'REJECT_CARDINALITY', 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'dep_hour': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Late': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 5, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'DROP', 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Boolean'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Dest': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 100, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'NONE', 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Text'}, 'autoReason': 'REJECT_CARDINALITY', 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'dep_dow': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'AirTime': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'UniqueCarrier': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 5, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'DROP', 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Text'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Late_avg': {'generate_derivative': False, 'impute_constant_value': 0.0, 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'TARGET', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'DoubleMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'dep_month': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'dep_woy': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Distance': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}}, 'reduce': {'enabled': False, 'kept_variance': 0.0}, 'feature_generation': {'pairwise_linear': {'behavior': 'DISABLED'}, 'polynomial_combinations': {'behavior': 'DISABLED'}, 'manual_interactions': {'interactions': []}, 'numericals_clustering': {'k': 0, 'all_features': False, 'input_features': [], 'behavior': 'DISABLED'}, 'categoricals_count_transformer': {'all_features': False, 'input_features': [], 'behavior': 'DISABLED'}}, 'feature_selection_params': {'method': 'NONE', 'random_forest_params': {'n_trees': 30, 'depth': 10, 'n_features': 25}, 'lasso_params': {'alpha': [0.01, 0.1, 1.0, 10.0, 100.0], 'cross_validate': True}, 'pca_params': {'n_features': 25, 'variance_proportion': 0.9}, 'correlation_params': {'min_abs_correlation': 0.0, 'max_abs_correlation': 1.0, 'n_features': 25}, 'custom_params': {'code': '# type your code here'}}, 'preprocessingFitSampleRatio': 1.0, 'preprocessingFitSampleSeed': 1337}
[2021-04-14 19:03:31,432] [7842/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Loading train set
[2021-04-14 19:03:31,433] [7842/MainThread] [INFO] [root] Reading with dtypes: None
[2021-04-14 19:03:31,433] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_month: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 19:03:31,433] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_woy: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 19:03:31,433] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_dow: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 19:03:31,433] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_hour: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 19:03:31,433] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for UniqueCarrier: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 19:03:31,433] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for FlightNum: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 19:03:31,433] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for AirTime: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 19:03:31,433] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Late: str (schema_type=boolean feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 19:03:31,433] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Origin: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 19:03:31,433] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Dest: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 19:03:31,433] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Distance: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 19:03:31,433] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Late_avg: <class 'numpy.float64'> (schema_type=double feature_type=NUMERIC feature_role=TARGET)
[2021-04-14 19:03:31,433] [7842/MainThread] [INFO] [root] Reading with FIXED dtypes: {'dep_month': <class 'numpy.float64'>, 'dep_woy': <class 'numpy.float64'>, 'dep_dow': <class 'numpy.float64'>, 'dep_hour': <class 'numpy.float64'>, 'UniqueCarrier': 'str', 'FlightNum': <class 'numpy.float64'>, 'AirTime': <class 'numpy.float64'>, 'Late': 'str', 'Origin': 'str', 'Dest': 'str', 'Distance': <class 'numpy.float64'>, 'Late_avg': <class 'numpy.float64'>}
[2021-04-14 19:03:31,573] [7842/MainThread] [INFO] [root] Loaded table
[2021-04-14 19:03:31,574] [7842/MainThread] [INFO] [dataiku.doctor.utils]  Coercion done
[2021-04-14 19:03:31,574] [7842/MainThread] [INFO] [dataiku.doctor.utils.split] Loaded train df: shape=(79912,12)
[2021-04-14 19:03:31,574] [7842/MainThread] [INFO] [dataiku.doctor.commands] Train col : dep_month (float64)
[2021-04-14 19:03:31,574] [7842/MainThread] [INFO] [dataiku.doctor.commands] Train col : dep_woy (float64)
[2021-04-14 19:03:31,574] [7842/MainThread] [INFO] [dataiku.doctor.commands] Train col : dep_dow (float64)
[2021-04-14 19:03:31,574] [7842/MainThread] [INFO] [dataiku.doctor.commands] Train col : dep_hour (float64)
[2021-04-14 19:03:31,574] [7842/MainThread] [INFO] [dataiku.doctor.commands] Train col : UniqueCarrier (object)
[2021-04-14 19:03:31,574] [7842/MainThread] [INFO] [dataiku.doctor.commands] Train col : FlightNum (float64)
[2021-04-14 19:03:31,574] [7842/MainThread] [INFO] [dataiku.doctor.commands] Train col : AirTime (float64)
[2021-04-14 19:03:31,574] [7842/MainThread] [INFO] [dataiku.doctor.commands] Train col : Late (object)
[2021-04-14 19:03:31,574] [7842/MainThread] [INFO] [dataiku.doctor.commands] Train col : Origin (object)
[2021-04-14 19:03:31,574] [7842/MainThread] [INFO] [dataiku.doctor.commands] Train col : Dest (object)
[2021-04-14 19:03:31,574] [7842/MainThread] [INFO] [dataiku.doctor.commands] Train col : Distance (float64)
[2021-04-14 19:03:31,574] [7842/MainThread] [INFO] [dataiku.doctor.commands] Train col : Late_avg (float64)
[2021-04-14 19:03:31,574] [7842/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Loading train set
[2021-04-14 19:03:31,574] [7842/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Loading test set
[2021-04-14 19:03:31,575] [7842/MainThread] [INFO] [root] Reading with dtypes: None
[2021-04-14 19:03:31,575] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_month: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 19:03:31,575] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_woy: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 19:03:31,575] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_dow: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 19:03:31,575] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_hour: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 19:03:31,575] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for UniqueCarrier: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 19:03:31,575] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for FlightNum: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 19:03:31,575] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for AirTime: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 19:03:31,575] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Late: str (schema_type=boolean feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 19:03:31,575] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Origin: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 19:03:31,575] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Dest: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 19:03:31,575] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Distance: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 19:03:31,575] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Late_avg: <class 'numpy.float64'> (schema_type=double feature_type=NUMERIC feature_role=TARGET)
[2021-04-14 19:03:31,575] [7842/MainThread] [INFO] [root] Reading with FIXED dtypes: {'dep_month': <class 'numpy.float64'>, 'dep_woy': <class 'numpy.float64'>, 'dep_dow': <class 'numpy.float64'>, 'dep_hour': <class 'numpy.float64'>, 'UniqueCarrier': 'str', 'FlightNum': <class 'numpy.float64'>, 'AirTime': <class 'numpy.float64'>, 'Late': 'str', 'Origin': 'str', 'Dest': 'str', 'Distance': <class 'numpy.float64'>, 'Late_avg': <class 'numpy.float64'>}
[2021-04-14 19:03:31,612] [7842/MainThread] [INFO] [root] Loaded table
[2021-04-14 19:03:31,612] [7842/MainThread] [INFO] [dataiku.doctor.utils]  Coercion done
[2021-04-14 19:03:31,612] [7842/MainThread] [INFO] [dataiku.doctor.utils.split] Loaded test df: shape=(20088,12)
[2021-04-14 19:03:31,613] [7842/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Loading test set
[2021-04-14 19:03:31,613] [7842/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Collecting statistics
[2021-04-14 19:03:31,688] [7842/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at FlightNum... (type=NUMERIC)
[2021-04-14 19:03:31,688] [7842/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-14 19:03:31,695] [7842/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Origin... (type=CATEGORY)
[2021-04-14 19:03:31,707] [7842/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at dep_hour... (type=NUMERIC)
[2021-04-14 19:03:31,707] [7842/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-14 19:03:31,711] [7842/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Late... (type=CATEGORY)
[2021-04-14 19:03:31,721] [7842/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Dest... (type=CATEGORY)
[2021-04-14 19:03:31,736] [7842/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at dep_dow... (type=NUMERIC)
[2021-04-14 19:03:31,736] [7842/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-14 19:03:31,740] [7842/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at AirTime... (type=NUMERIC)
[2021-04-14 19:03:31,741] [7842/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-14 19:03:31,745] [7842/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at UniqueCarrier... (type=CATEGORY)
[2021-04-14 19:03:31,755] [7842/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Late_avg... (type=NUMERIC)
[2021-04-14 19:03:31,755] [7842/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at dep_month... (type=NUMERIC)
[2021-04-14 19:03:31,755] [7842/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-14 19:03:31,759] [7842/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at dep_woy... (type=NUMERIC)
[2021-04-14 19:03:31,759] [7842/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-14 19:03:31,762] [7842/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Distance... (type=NUMERIC)
[2021-04-14 19:03:31,762] [7842/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-14 19:03:31,766] [7842/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Collecting statistics
[2021-04-14 19:03:31,766] [7842/MainThread] [INFO] [dataiku.doctor.multiframe] generating interactions
[2021-04-14 19:03:31,766] [7842/MainThread] [INFO] [dataiku.doctor.multiframe] {'target_remapping': [], 'skipPreprocessing': False, 'per_feature': {'FlightNum': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Origin': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 100, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'NONE', 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Text'}, 'autoReason': 'REJECT_CARDINALITY', 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'dep_hour': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Late': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 5, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'DROP', 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Boolean'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Dest': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 100, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'NONE', 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Text'}, 'autoReason': 'REJECT_CARDINALITY', 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'dep_dow': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'AirTime': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'UniqueCarrier': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 5, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'DROP', 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Text'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Late_avg': {'generate_derivative': False, 'impute_constant_value': 0.0, 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'TARGET', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'DoubleMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'dep_month': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'dep_woy': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Distance': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'rescaling': 'AVGSTD', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}}, 'reduce': {'enabled': False, 'kept_variance': 0.0}, 'feature_generation': {'pairwise_linear': {'behavior': 'DISABLED'}, 'polynomial_combinations': {'behavior': 'DISABLED'}, 'manual_interactions': {'interactions': []}, 'numericals_clustering': {'k': 0, 'all_features': False, 'input_features': [], 'behavior': 'DISABLED'}, 'categoricals_count_transformer': {'all_features': False, 'input_features': [], 'behavior': 'DISABLED'}}, 'feature_selection_params': {'method': 'NONE', 'random_forest_params': {'n_trees': 30, 'depth': 10, 'n_features': 25}, 'lasso_params': {'alpha': [0.01, 0.1, 1.0, 10.0, 100.0], 'cross_validate': True}, 'pca_params': {'n_features': 25, 'variance_proportion': 0.9}, 'correlation_params': {'min_abs_correlation': 0.0, 'max_abs_correlation': 1.0, 'n_features': 25}, 'custom_params': {'code': '# type your code here'}}, 'preprocessingFitSampleRatio': 1.0, 'preprocessingFitSampleSeed': 1337}
[2021-04-14 19:03:31,766] [7842/MainThread] [INFO] [dataiku.doctor.multiframe] No feature selection to perform
[2021-04-14 19:03:31,766] [7842/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Preprocessing train set
[2021-04-14 19:03:31,767] [7842/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 79912
[2021-04-14 19:03:31,767] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RemapValueToOutput
[2021-04-14 19:03:31,767] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:MultipleImputeMissingFromInput
[2021-04-14 19:03:31,767] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {'FlightNum': 2149.660088597457, 'dep_hour': 13.086732906196817, 'dep_dow': 3.9422364601061166, 'AirTime': 103.37577585343878, 'dep_month': 3.7664931424567025, 'dep_woy': 14.544386324957452, 'Distance': 724.2432175392933}
[2021-04-14 19:03:31,772] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (FlightNum)
[2021-04-14 19:03:31,773] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale FlightNum (avg=2149.660088597457 std=1944.0927463213632 shift=2149.660088597457 inv_scale=0.0005143787516784951)
[2021-04-14 19:03:31,775] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled FlightNum (avg=3.698891006059186e-17 std=1.0) nulls=0
[2021-04-14 19:03:31,775] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (dep_hour)
[2021-04-14 19:03:31,776] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_hour (avg=13.086732906196817 std=4.80188247321774 shift=13.086732906196817 inv_scale=0.20825166079708327)
[2021-04-14 19:03:31,778] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_hour (avg=-4.4813487188793987e-17 std=0.9999999999999999) nulls=0
[2021-04-14 19:03:31,778] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (dep_dow)
[2021-04-14 19:03:31,779] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_dow (avg=3.9422364601061166 std=1.9826725071034723 shift=3.9422364601061166 inv_scale=0.5043697314696318)
[2021-04-14 19:03:31,781] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_dow (avg=9.104962476453381e-17 std=1.0000000000000002) nulls=0
[2021-04-14 19:03:31,781] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (AirTime)
[2021-04-14 19:03:31,782] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale AirTime (avg=103.37577585343878 std=68.48571355586222 shift=103.37577585343878 inv_scale=0.01460158547058611)
[2021-04-14 19:03:31,783] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled AirTime (avg=2.1339755804187613e-17 std=1.0) nulls=0
[2021-04-14 19:03:31,784] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (dep_month)
[2021-04-14 19:03:31,784] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_month (avg=3.7664931424567025 std=1.5919104244248008 shift=3.7664931424567025 inv_scale=0.6281760485118542)
[2021-04-14 19:03:31,786] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_month (avg=-6.828721857340036e-17 std=1.0) nulls=0
[2021-04-14 19:03:31,786] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (dep_woy)
[2021-04-14 19:03:31,787] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_woy (avg=14.544386324957452 std=6.971213818152594 shift=14.544386324957452 inv_scale=0.14344704180440773)
[2021-04-14 19:03:31,789] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_woy (avg=2.2762406191133452e-17 std=1.0) nulls=0
[2021-04-14 19:03:31,789] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (Distance)
[2021-04-14 19:03:31,790] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale Distance (avg=724.2432175392933 std=561.8104116867005 shift=724.2432175392933 inv_scale=0.0017799598925155923)
[2021-04-14 19:03:31,791] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled Distance (avg=-5.690601547783363e-17 std=0.9999999999999999) nulls=0
[2021-04-14 19:03:31,791] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FlushDFBuilder(num_flagonly)
[2021-04-14 19:03:31,791] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FastSparseDummifyProcessor (Origin)
[2021-04-14 19:03:31,814] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(79912, 103) nnz=79912
[2021-04-14 19:03:31,815] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FastSparseDummifyProcessor (Late)
[2021-04-14 19:03:31,833] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(79912, 2) nnz=60354
[2021-04-14 19:03:31,834] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FastSparseDummifyProcessor (Dest)
[2021-04-14 19:03:31,856] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(79912, 103) nnz=79912
[2021-04-14 19:03:31,857] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FastSparseDummifyProcessor (UniqueCarrier)
[2021-04-14 19:03:31,875] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(79912, 7) nnz=41098
[2021-04-14 19:03:31,876] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:MultipleImputeMissingFromInput
[2021-04-14 19:03:31,876] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2021-04-14 19:03:31,876] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FlushDFBuilder(cat_flagpresence)
[2021-04-14 19:03:31,876] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:MultipleImputeMissingFromInput
[2021-04-14 19:03:31,876] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2021-04-14 19:03:31,876] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FlushDFBuilder(interaction)
[2021-04-14 19:03:31,876] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RealignTarget
[2021-04-14 19:03:31,876] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] Realign target series = (79912,)
[2021-04-14 19:03:31,878] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] After realign target: (79912,)
[2021-04-14 19:03:31,878] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:DropRowsWhereNoTarget
[2021-04-14 19:03:31,878] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] Deleting 0 rows because one of ['target'] is missing
[2021-04-14 19:03:31,878] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] MF before = (79912, 222)
[2021-04-14 19:03:31,878] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] target before = (79912,)
[2021-04-14 19:03:31,880] [7842/MainThread] [INFO] [dataiku.doctor.multiframe] MultiFrame, dropping rows: []
[2021-04-14 19:03:31,904] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] After DRWNT input_df=(79912, 12)
[2021-04-14 19:03:31,904] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] MF after = (79912, 222)
[2021-04-14 19:03:31,904] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] target after = (79912,)
[2021-04-14 19:03:31,904] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:DumpPipelineState
[2021-04-14 19:03:31,904] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] ********* Pipeline state (Before feature selection)
[2021-04-14 19:03:31,904] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]    input_df= (79912, 12) 
[2021-04-14 19:03:31,904] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]    current_mf=(79912, 222) 
[2021-04-14 19:03:31,904] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]    PPR: 
[2021-04-14 19:03:31,904] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]       target = <class 'pandas.core.series.Series'> ((79912,))
[2021-04-14 19:03:31,904] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:EmitCurrentMFAsResult
[2021-04-14 19:03:31,904] [7842/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 79912
[2021-04-14 19:03:31,904] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:DumpPipelineState
[2021-04-14 19:03:31,904] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] ********* Pipeline state (At end)
[2021-04-14 19:03:31,905] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]    input_df= (79912, 12) 
[2021-04-14 19:03:31,905] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]    current_mf=(0, 0) 
[2021-04-14 19:03:31,905] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]    PPR: 
[2021-04-14 19:03:31,905] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]       target = <class 'pandas.core.series.Series'> ((79912,))
[2021-04-14 19:03:31,905] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]       TRAIN = <class 'dataiku.doctor.multiframe.MultiFrame'> ((79912, 222))
[2021-04-14 19:03:31,905] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]       UNPROCESSED = <class 'pandas.core.frame.DataFrame'> ((79912, 12))
[2021-04-14 19:03:31,949] [7842/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Preprocessing train set
[2021-04-14 19:03:31,949] [7842/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Preprocessing test set
[2021-04-14 19:03:31,950] [7842/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 20088
[2021-04-14 19:03:31,950] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RemapValueToOutput
[2021-04-14 19:03:31,950] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:MultipleImputeMissingFromInput
[2021-04-14 19:03:31,950] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {'FlightNum': 2149.660088597457, 'dep_hour': 13.086732906196817, 'dep_dow': 3.9422364601061166, 'AirTime': 103.37577585343878, 'dep_month': 3.7664931424567025, 'dep_woy': 14.544386324957452, 'Distance': 724.2432175392933}
[2021-04-14 19:03:31,952] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (FlightNum)
[2021-04-14 19:03:31,953] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale FlightNum (avg=2129.2614496216647 std=1938.8000573324396 shift=2149.660088597457 inv_scale=0.0005143787516784951)
[2021-04-14 19:03:31,954] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled FlightNum (avg=-0.010492626452308425 std=0.997277553244855) nulls=0
[2021-04-14 19:03:31,954] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (dep_hour)
[2021-04-14 19:03:31,954] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_hour (avg=13.137544802867383 std=4.811521844053203 shift=13.086732906196817 inv_scale=0.20825166079708327)
[2021-04-14 19:03:31,955] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_hour (avg=0.01058166186989532 std=1.002007414985524) nulls=0
[2021-04-14 19:03:31,955] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (dep_dow)
[2021-04-14 19:03:31,956] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_dow (avg=3.9275189167662288 std=1.9835343957941927 shift=3.9422364601061166 inv_scale=0.5043697314696318)
[2021-04-14 19:03:31,957] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_dow (avg=-0.007423083382231956 std=1.0004347105674953) nulls=0
[2021-04-14 19:03:31,957] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (AirTime)
[2021-04-14 19:03:31,957] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale AirTime (avg=103.51359020310633 std=68.67809152402833 shift=103.37577585343878 inv_scale=0.01460158547058611)
[2021-04-14 19:03:31,958] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled AirTime (avg=0.002012308005743982 std=1.0028090233448352) nulls=0
[2021-04-14 19:03:31,958] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (dep_month)
[2021-04-14 19:03:31,959] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_month (avg=3.759856630824373 std=1.5818950705067425 shift=3.7664931424567025 inv_scale=0.6281760485118542)
[2021-04-14 19:03:31,960] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_month (avg=-0.0041688976530997466 std=0.9937085945513064) nulls=0
[2021-04-14 19:03:31,960] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (dep_woy)
[2021-04-14 19:03:31,960] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_woy (avg=14.508910792512943 std=6.9378626117674775 shift=14.544386324957452 inv_scale=0.14344704180440773)
[2021-04-14 19:03:31,961] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_woy (avg=-0.005088860185601145 std=0.9952158681034469) nulls=0
[2021-04-14 19:03:31,961] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (Distance)
[2021-04-14 19:03:31,962] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale Distance (avg=726.2480585424133 std=562.197650726872 shift=724.2432175392933 inv_scale=0.0017799598925155923)
[2021-04-14 19:03:31,963] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled Distance (avg=0.0035685365764245467 std=1.0006892699603216) nulls=0
[2021-04-14 19:03:31,963] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FlushDFBuilder(num_flagonly)
[2021-04-14 19:03:31,963] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FastSparseDummifyProcessor (Origin)
[2021-04-14 19:03:31,969] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(20088, 103) nnz=20088
[2021-04-14 19:03:31,969] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FastSparseDummifyProcessor (Late)
[2021-04-14 19:03:31,975] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(20088, 2) nnz=15203
[2021-04-14 19:03:31,975] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FastSparseDummifyProcessor (Dest)
[2021-04-14 19:03:31,982] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(20088, 103) nnz=20088
[2021-04-14 19:03:31,982] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FastSparseDummifyProcessor (UniqueCarrier)
[2021-04-14 19:03:31,987] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(20088, 7) nnz=10425
[2021-04-14 19:03:31,987] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:MultipleImputeMissingFromInput
[2021-04-14 19:03:31,987] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2021-04-14 19:03:31,987] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FlushDFBuilder(cat_flagpresence)
[2021-04-14 19:03:31,987] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:MultipleImputeMissingFromInput
[2021-04-14 19:03:31,987] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2021-04-14 19:03:31,988] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FlushDFBuilder(interaction)
[2021-04-14 19:03:31,988] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RealignTarget
[2021-04-14 19:03:31,988] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] Realign target series = (20088,)
[2021-04-14 19:03:31,988] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] After realign target: (20088,)
[2021-04-14 19:03:31,988] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:DropRowsWhereNoTarget
[2021-04-14 19:03:31,988] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] Deleting 0 rows because one of ['target'] is missing
[2021-04-14 19:03:31,988] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] MF before = (20088, 222)
[2021-04-14 19:03:31,989] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] target before = (20088,)
[2021-04-14 19:03:31,989] [7842/MainThread] [INFO] [dataiku.doctor.multiframe] MultiFrame, dropping rows: []
[2021-04-14 19:03:31,996] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] After DRWNT input_df=(20088, 12)
[2021-04-14 19:03:31,996] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] MF after = (20088, 222)
[2021-04-14 19:03:31,996] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] target after = (20088,)
[2021-04-14 19:03:31,996] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:DumpPipelineState
[2021-04-14 19:03:31,996] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] ********* Pipeline state (Before feature selection)
[2021-04-14 19:03:31,997] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]    input_df= (20088, 12) 
[2021-04-14 19:03:31,997] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]    current_mf=(20088, 222) 
[2021-04-14 19:03:31,997] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]    PPR: 
[2021-04-14 19:03:31,997] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]       target = <class 'pandas.core.series.Series'> ((20088,))
[2021-04-14 19:03:31,997] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:EmitCurrentMFAsResult
[2021-04-14 19:03:31,997] [7842/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 20088
[2021-04-14 19:03:31,997] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:DumpPipelineState
[2021-04-14 19:03:31,997] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] ********* Pipeline state (At end)
[2021-04-14 19:03:31,997] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]    input_df= (20088, 12) 
[2021-04-14 19:03:31,997] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]    current_mf=(0, 0) 
[2021-04-14 19:03:31,997] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]    PPR: 
[2021-04-14 19:03:31,997] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]       target = <class 'pandas.core.series.Series'> ((20088,))
[2021-04-14 19:03:31,997] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]       TRAIN = <class 'dataiku.doctor.multiframe.MultiFrame'> ((20088, 222))
[2021-04-14 19:03:31,997] [7842/MainThread] [DEBUG] [dku.ml.preprocessing]       UNPROCESSED = <class 'pandas.core.frame.DataFrame'> ((20088, 12))
[2021-04-14 19:03:31,997] [7842/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Preprocessing test set
[2021-04-14 19:03:32,050] [7842/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Hyperparameter searching
[2021-04-14 19:03:32,097] [7842/MainThread] [DEBUG] [dku.ml.preprocessing] Deleting 0 rows
[2021-04-14 19:03:32,097] [7842/MainThread] [INFO] [dataiku.doctor.multiframe] MultiFrame, dropping rows: []
[2021-04-14 19:03:32,110] [7842/MainThread] [INFO] [dataiku.doctor.prediction.common] prepare multiframe shape=(79912,222) tn=17740464 nnz=820660 fill_ratio=0.05
[2021-04-14 19:03:32,110] [7842/MainThread] [INFO] [dataiku.doctor.prediction.common] too small, using array
[2021-04-14 19:03:32,215] [7842/MainThread] [INFO] [dataiku.doctor.prediction.common] Create CLF from params: {'algorithm': 'RIDGE_REGRESSION', 'ridge_grid': {'alphaMode': 'MANUAL', 'alpha': {'gridMode': 'EXPLICIT', 'randomMode': 'RANGE', 'limit': {'min': 1e-23}, 'values': [0.1, 1.0, 3.0], 'range': {'min': 0.1, 'max': 3.0, 'nbValues': 3, 'scaling': 'LOGARITHMIC'}}, 'enabled': True}, 'max_ensemble_nodes_serialized': 0, 'metrics': {'evaluationMetric': 'R2', 'customEvaluationMetricGIB': True, 'customEvaluationMetricNeedsProba': False, 'costMatrixWeights': {'tpGain': 1.0, 'tnGain': 0.0, 'fpGain': -0.3, 'fnGain': 0.0}, 'liftPoint': 0.4}, 'autoOptimizeThreshold': True, 'forcedClassifierThreshold': 0.0, 'gridLength': 3, 'grid_search_params': {'mode': 'KFOLD', 'splitRatio': 0.8, 'shuffleIterations': 1, 'nFolds': 5, 'stratified': True, 'strategy': 'GRID', 'bayesianOptimizer': 'SCIKIT_OPTIMIZE', 'randomized': True, 'seed': 0, 'nIter': 0, 'nIterRandom': 3, 'timeout': 0, 'nJobs': 4, 'distributed': False, 'nContainers': 4}, 'pluginAlgoCustomGridSearch': False, 'computeLearningCurves': False, 'skipExpensiveReports': False} for algorithm RIDGE_REGRESSION
[2021-04-14 19:03:32,215] [7842/MainThread] [INFO] [dataiku.doctor.prediction.common] Using K-Fold CV with k=5
[2021-04-14 19:03:32,220] [7842/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluation_monitor] No distributed container configuration is available to run this search
[2021-04-14 19:03:32,225] [7842/MainThread] [INFO] [dataiku.doctor.crossval.search_runner] Fitting 5 folds for each of 3 candidates, totalling 15 fits
[2021-04-14 19:03:32,225] [7842/MainThread] [INFO] [dataiku.doctor.crossval.search_runner] Execute hyperparameter search locally on 4 threads
[2021-04-14 19:03:32,225] [7842/MainThread] [INFO] [dataiku.doctor.crossval.strategies.grid_search_strategy] Running GridSearchStrategy for hyperparameters space: <dataiku.doctor.prediction.common.HyperparametersSpace object at 0x7f838882ea20>
[2021-04-14 19:03:32,225] [7842/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 0: {'alpha': 0.1}
[2021-04-14 19:03:32,225] [7842/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.local_worker] Starting worker: local-0
[2021-04-14 19:03:32,225] [7842/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.local_worker] Started worker: local-0
[2021-04-14 19:03:32,226] [7842/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 1: {'alpha': 0.1}
[2021-04-14 19:03:32,226] [7842/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 19:03:32,226] [7842/Thread-2:local-0] [INFO] [dataiku.doctor.crossval.search_context] Fit s=0: alpha=0.1 .......................................................
[2021-04-14 19:03:32,226] [7842/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.local_worker] Starting worker: local-1
[2021-04-14 19:03:32,226] [7842/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 2: {'alpha': 0.1}
[2021-04-14 19:03:32,226] [7842/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.local_worker] Started worker: local-1
[2021-04-14 19:03:32,226] [7842/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.local_worker] Starting worker: local-2
[2021-04-14 19:03:32,226] [7842/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 19:03:32,226] [7842/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 3: {'alpha': 0.1}
[2021-04-14 19:03:32,226] [7842/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.local_worker] Started worker: local-2
[2021-04-14 19:03:32,227] [7842/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Fit s=1: alpha=0.1 .......................................................
[2021-04-14 19:03:32,227] [7842/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.local_worker] Starting worker: local-3
[2021-04-14 19:03:32,227] [7842/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 19:03:32,227] [7842/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 4: {'alpha': 0.1}
[2021-04-14 19:03:32,227] [7842/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.local_worker] Started worker: local-3
[2021-04-14 19:03:32,227] [7842/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Fit s=2: alpha=0.1 .......................................................
[2021-04-14 19:03:32,228] [7842/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 19:03:32,228] [7842/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Fit s=3: alpha=0.1 .......................................................
[2021-04-14 19:03:32,886] [7842/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Done s=3: alpha=0.1 (ft=0.6s st=0.1s sc=0.8449398620182682, sg=1)
[2021-04-14 19:03:32,906] [7842/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Done s=1: alpha=0.1 (ft=0.6s st=0.1s sc=0.8538902972183009, sg=1)
[2021-04-14 19:03:32,924] [7842/Thread-2:local-0] [INFO] [dataiku.doctor.crossval.search_context] Done s=0: alpha=0.1 (ft=0.7s st=0.0s sc=0.8457393757743862, sg=1)
[2021-04-14 19:03:32,939] [7842/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 19:03:32,939] [7842/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 19:03:32,940] [7842/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Fit s=4: alpha=0.1 .......................................................
[2021-04-14 19:03:32,946] [7842/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Done s=2: alpha=0.1 (ft=0.6s st=0.1s sc=0.8490534413428655, sg=1)
[2021-04-14 19:03:32,940] [7842/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 19:03:32,960] [7842/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 0: {'alpha': 3.0}
[2021-04-14 19:03:32,961] [7842/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 1: {'alpha': 3.0}
[2021-04-14 19:03:32,961] [7842/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 19:03:32,961] [7842/Thread-2:local-0] [INFO] [dataiku.doctor.crossval.search_context] Fit s=0: alpha=3.0 .......................................................
[2021-04-14 19:03:32,966] [7842/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 19:03:32,970] [7842/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 19:03:32,970] [7842/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 19:03:32,975] [7842/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Fit s=1: alpha=3.0 .......................................................
[2021-04-14 19:03:32,986] [7842/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 2: {'alpha': 3.0}
[2021-04-14 19:03:32,986] [7842/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 19:03:32,986] [7842/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Fit s=2: alpha=3.0 .......................................................
[2021-04-14 19:03:32,988] [7842/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 3: {'alpha': 3.0}
[2021-04-14 19:03:33,625] [7842/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Done s=4: alpha=0.1 (ft=0.6s st=0.0s sc=0.8457860195038889, sg=1)
[2021-04-14 19:03:33,641] [7842/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 19:03:33,642] [7842/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Done s=1: alpha=3.0 (ft=0.6s st=0.1s sc=0.43234126047484334, sg=1)
[2021-04-14 19:03:33,654] [7842/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 19:03:33,655] [7842/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 19:03:33,661] [7842/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Fit s=3: alpha=3.0 .......................................................
[2021-04-14 19:03:33,666] [7842/Thread-2:local-0] [INFO] [dataiku.doctor.crossval.search_context] Done s=0: alpha=3.0 (ft=0.6s st=0.1s sc=0.4288827638895817, sg=1)
[2021-04-14 19:03:33,678] [7842/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Done s=2: alpha=3.0 (ft=0.6s st=0.1s sc=0.43243440504479735, sg=1)
[2021-04-14 19:03:33,689] [7842/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 19:03:33,689] [7842/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 19:03:33,941] [7842/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Done s=3: alpha=3.0 (ft=0.2s st=0.0s sc=0.4290874817540169, sg=1)
[2021-04-14 19:03:33,961] [7842/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 19:03:34,521] [7842/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 4: {'alpha': 3.0}
[2021-04-14 19:03:34,521] [7842/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 19:03:34,521] [7842/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Fit s=4: alpha=3.0 .......................................................
[2021-04-14 19:03:34,521] [7842/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 0: {'alpha': 1.0}
[2021-04-14 19:03:34,522] [7842/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 19:03:34,522] [7842/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Fit s=0: alpha=1.0 .......................................................
[2021-04-14 19:03:34,522] [7842/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 1: {'alpha': 1.0}
[2021-04-14 19:03:34,522] [7842/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 19:03:34,522] [7842/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Fit s=1: alpha=1.0 .......................................................
[2021-04-14 19:03:34,523] [7842/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 2: {'alpha': 1.0}
[2021-04-14 19:03:34,523] [7842/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 19:03:34,523] [7842/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 3: {'alpha': 1.0}
[2021-04-14 19:03:34,523] [7842/Thread-2:local-0] [INFO] [dataiku.doctor.crossval.search_context] Fit s=2: alpha=1.0 .......................................................
[2021-04-14 19:03:35,220] [7842/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Done s=1: alpha=1.0 (ft=0.6s st=0.1s sc=0.675994089889711, sg=1)
[2021-04-14 19:03:35,238] [7842/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Done s=0: alpha=1.0 (ft=0.7s st=0.0s sc=0.669476675230695, sg=1)
[2021-04-14 19:03:35,250] [7842/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 19:03:35,251] [7842/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 4: {'alpha': 1.0}
[2021-04-14 19:03:35,251] [7842/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 19:03:35,260] [7842/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 19:03:35,270] [7842/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task...
[2021-04-14 19:03:35,270] [7842/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Fit s=4: alpha=1.0 .......................................................
[2021-04-14 19:03:35,272] [7842/Thread-2:local-0] [INFO] [dataiku.doctor.crossval.search_context] Done s=2: alpha=1.0 (ft=0.7s st=0.1s sc=0.6738472683255516, sg=1)
[2021-04-14 19:03:35,270] [7842/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Fit s=3: alpha=1.0 .......................................................
[2021-04-14 19:03:35,292] [7842/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Done s=4: alpha=3.0 (ft=0.7s st=0.1s sc=0.4297451224713805, sg=1)
[2021-04-14 19:03:35,313] [7842/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 19:03:35,313] [7842/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 19:03:35,618] [7842/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Done s=3: alpha=1.0 (ft=0.3s st=0.0s sc=0.6695676145042324, sg=1)
[2021-04-14 19:03:35,634] [7842/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 19:03:35,646] [7842/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Done s=4: alpha=1.0 (ft=0.3s st=0.1s sc=0.6709521607598198, sg=1)
[2021-04-14 19:03:35,659] [7842/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Task done
[2021-04-14 19:03:36,234] [7842/Thread-1] [INFO] [dataiku.doctor.crossval.search_runner] Completed search for hyperparameters
[2021-04-14 19:03:36,234] [7842/Thread-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Scheduler has been soft interrupted
[2021-04-14 19:03:36,234] [7842/MainThread] [INFO] [dataiku.doctor.distributed.work_scheduler] Scheduler has been hard interrupted (shutdown)
[2021-04-14 19:03:36,234] [7842/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.local_worker] Stopping worker: local-0
[2021-04-14 19:03:36,234] [7842/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.local_worker] Stopped worker: local-0
[2021-04-14 19:03:36,234] [7842/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.local_worker] Stopping worker: local-2
[2021-04-14 19:03:36,235] [7842/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.local_worker] Stopped worker: local-2
[2021-04-14 19:03:36,235] [7842/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.local_worker] Stopping worker: local-1
[2021-04-14 19:03:36,235] [7842/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.local_worker] Stopped worker: local-1
[2021-04-14 19:03:36,235] [7842/Thread-5:local-3] [ERROR] [dataiku.doctor.distributed.work_scheduler] All workers are dead, interrupt the scheduler
[2021-04-14 19:03:36,235] [7842/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.local_worker] Stopping worker: local-3
[2021-04-14 19:03:36,235] [7842/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.local_worker] Stopped worker: local-3
[2021-04-14 19:03:36,235] [7842/MainThread] [INFO] [dataiku.doctor.crossval.search_runner] Hyperparameter search done, best_parameters being : {'alpha': 0.1}
[2021-04-14 19:03:36,235] [7842/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Hyperparameter searching
[2021-04-14 19:03:36,754] [7842/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Fitting model
[2021-04-14 19:03:36,803] [7842/MainThread] [INFO] [dataiku.doctor.prediction.common] Fitting model:
[2021-04-14 19:03:36,804] [7842/MainThread] [INFO] [dataiku.doctor.prediction.common]   Model is: Ridge(alpha=0.1, copy_X=True, fit_intercept=True, max_iter=None,
   normalize=True, random_state=None, solver='auto', tol=0.001)
[2021-04-14 19:03:36,804] [7842/MainThread] [INFO] [dataiku.doctor.prediction.common]   train_X class: <class 'numpy.ndarray'>
[2021-04-14 19:03:36,804] [7842/MainThread] [INFO] [dataiku.doctor.prediction.common]   train_X shape: (79912, 222)
[2021-04-14 19:03:36,804] [7842/MainThread] [INFO] [dataiku.doctor.prediction.common]   train_y shape: (79912,)
[2021-04-14 19:03:36,804] [7842/MainThread] [INFO] [dataiku.doctor.prediction.common]   calibration enabled: a sub-sample of the train data has been saved for calibration
[2021-04-14 19:03:37,014] [7842/MainThread] [INFO] [dataiku.doctor.prediction.regression_fit] Output params are {'resolved': {'algorithm': 'RIDGE_REGRESSION', 'skipExpensiveReports': False, 'ridge': {'alpha': 0.1}}, 'other': {}}
[2021-04-14 19:03:37,014] [7842/MainThread] [INFO] [dataiku.doctor.prediction.scoring_base] No feature importance in classifier
[2021-04-14 19:03:37,015] [7842/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Fitting model
[2021-04-14 19:03:37,015] [7842/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Saving model
[2021-04-14 19:03:37,016] [7842/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Saving model
[2021-04-14 19:03:37,016] [7842/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Scoring model
[2021-04-14 19:03:37,633] [7842/MainThread] [INFO] [dataiku.doctor.prediction.regression_scoring] Intrinsic scoring
[2021-04-14 19:03:37,634] [7842/MainThread] [INFO] [dataiku.doctor.prediction.regression_scoring] Extracting rescalers
[2021-04-14 19:03:37,634] [7842/MainThread] [INFO] [dataiku.doctor.prediction.scoring_base] Computing feature importance
[2021-04-14 19:03:37,634] [7842/MainThread] [INFO] [dataiku.doctor.prediction.scoring_base] No feature importance in classifier
[2021-04-14 19:03:37,634] [7842/MainThread] [INFO] [dataiku.doctor.prediction.regression_scoring] Computing regression coeffs
[2021-04-14 19:03:37,777] [7842/MainThread] [INFO] [dataiku.doctor.prediction.scoring_base] Coefs: [ 2.51469332e-01  1.63509404e-03  2.21563661e-04 -1.08438658e-04
  7.74309566e-04 -2.14135594e-03 -2.34731325e-03  4.05513302e-04
 -3.84224483e-03  9.98158460e-02  9.73352511e-03  3.31158614e-02
 -2.11510611e-02 -5.19063094e-02 -3.50484037e-03  7.58730492e-02
  1.87917073e-02  8.09058303e-03  1.07808991e-01 -1.42025066e-03
 -3.57123318e-02  8.00747399e-03  1.66052964e-02  1.25769508e-01
  3.87993505e-02  1.02901567e-01 -1.64331156e-02  2.74977094e-02
 -1.08664556e-02 -3.45798688e-02  9.45740161e-02  3.84245341e-02
  4.07675482e-02 -7.42267692e-02 -3.20179030e-02  3.11095285e-02
 -5.55087529e-02 -1.26314355e-01  3.40451396e-02 -8.33427970e-02
  6.34162347e-03 -6.44124595e-02 -6.80267451e-02 -1.32847284e-02
 -1.43999553e-02 -2.19698199e-02 -7.09643456e-02 -8.18525041e-02
 -5.25408099e-02 -4.11485849e-02 -1.83727132e-02  5.17117170e-02
 -5.09081170e-02 -3.42249204e-02 -7.03537148e-02 -8.67745154e-03
 -3.42042142e-02 -8.35265271e-02 -5.80790945e-03  1.31546007e-02
  1.52431821e-05 -7.14909469e-02 -1.76890406e-02 -6.18477912e-02
  2.93530349e-02 -4.81405730e-02 -6.29962315e-02 -2.53277467e-02
 -1.22505796e-01 -3.71036993e-02 -2.98264254e-03 -4.39029431e-02
 -4.32659362e-02 -4.08451428e-02 -2.35020194e-02 -3.80725572e-02
 -1.00058510e-02 -8.49418930e-02  7.68226921e-02  1.91134938e-02
 -5.51348656e-02 -2.60788252e-02 -1.26555785e-02 -1.10936365e-02
 -5.09018549e-02 -1.12611236e-02  9.65819090e-03 -1.18088341e-01
 -9.06406762e-02 -3.15950205e-02 -8.76668154e-02 -3.94059781e-02
 -7.21306777e-02 -5.47178272e-03 -1.17713620e-01 -7.19217366e-03
 -2.40140346e-02 -4.26262830e-02  1.09233151e-01 -3.88513090e-02
 -3.01049878e-02 -9.12018091e-03  1.76993949e-02 -4.33849341e-02
 -1.50910624e-02  5.36356829e-02 -8.30521621e-02 -9.53223882e-02
 -9.20811049e-03  0.00000000e+00 -3.24684595e-02 -5.56575564e-03
  0.00000000e+00  4.75576909e-03  4.32262494e-03 -2.86530591e-03
 -4.64573117e-03 -9.32227539e-03 -6.89112886e-04 -4.95519343e-03
  2.26771083e-03 -3.13123599e-03 -2.32389800e-03  8.18470544e-04
  4.66506757e-03 -1.37985824e-02 -4.01609863e-03  4.24942309e-03
 -7.24107016e-04  2.87219361e-03  5.66910899e-03 -1.88568513e-03
  1.54078413e-03  3.64628176e-03  1.48072790e-03  2.62869307e-03
  4.20827592e-03 -1.07439927e-03  1.24574167e-03  3.15160083e-03
  1.35274917e-03  5.18687989e-03  1.52483718e-03  4.68810540e-03
 -3.24950307e-03 -4.64729168e-02  1.84114868e-03 -3.73831883e-03
  5.92466517e-03  5.15693211e-03  5.96984770e-04 -2.21766570e-03
 -3.66906822e-03 -2.29490588e-03  1.52302188e-03  8.18906536e-04
 -3.87318318e-03  2.28162375e-03  5.88795753e-03  1.72018987e-03
 -3.29947956e-03  3.64051904e-03  2.81195950e-03  7.10490409e-03
 -1.25385160e-03  5.90594337e-03  2.95074233e-03  6.43181124e-03
 -5.64077423e-04  7.61913752e-03 -2.19629505e-02  6.73559783e-03
  1.98083604e-03 -2.19535312e-03  2.16464275e-03  4.31880406e-03
  4.88183979e-04  5.28669474e-03  2.08017592e-03 -2.42593590e-03
  9.06516547e-03  3.14693881e-03  9.34723871e-04  7.01639450e-03
  6.29829625e-03  8.48767241e-03 -2.68895548e-02  2.57829856e-03
  2.85306587e-03 -2.22267703e-02 -3.73465546e-03  5.98110690e-03
  5.70350899e-03  1.77794837e-03 -1.78132240e-03  2.07376526e-04
  6.07618794e-03 -1.83115172e-03  2.08935910e-03  4.04539004e-03
 -4.07757400e-05  4.06332876e-03 -6.82279558e-04  5.27112271e-03
  5.83756884e-03  6.96279724e-04  4.45258371e-03  1.99413364e-03
  3.68748616e-03  4.03704078e-03  2.83384309e-03  6.47040973e-03
  9.93450521e-03  2.59075846e-03  0.00000000e+00  1.88719506e-03
 -4.23056143e-03 -7.18683236e-04 -2.19331330e-03 -1.14283550e-02
  3.40048855e-05  7.52671568e-04  0.00000000e+00]
[2021-04-14 19:03:37,777] [7842/MainThread] [INFO] [dataiku.doctor.prediction.scoring_base] Sample variance: 0.0006883796472037912
[2021-04-14 19:03:37,903] [7842/MainThread] [INFO] [dataiku.doctor.prediction.scoring_base] Singular variance matrix
[2021-04-14 19:03:37,918] [7842/MainThread] [INFO] [dataiku.doctor.prediction.common] prepare multiframe shape=(20088,222) tn=4459536 nnz=206420 fill_ratio=0.05
[2021-04-14 19:03:37,918] [7842/MainThread] [INFO] [dataiku.doctor.prediction.common] too small, using array
[2021-04-14 19:03:37,957] [7842/MainThread] [INFO] [dataiku.doctor.prediction.regression_scoring] Creating predictions on test set
[2021-04-14 19:03:37,967] [7842/MainThread] [INFO] [dataiku.doctor.prediction.regression_scoring] Computing regression performance on [0.24861573 0.24861573 0.24861573 ... 0.22897359 0.23421346 0.24112227]
[2021-04-14 19:03:39,075] [7842/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Scoring model
[2021-04-14 19:03:39,075] [7842/MainThread] [INFO] [root] Reading with dtypes: None
[2021-04-14 19:03:39,075] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_month: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 19:03:39,075] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_woy: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 19:03:39,075] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_dow: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 19:03:39,075] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_hour: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 19:03:39,075] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for UniqueCarrier: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 19:03:39,075] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for FlightNum: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 19:03:39,075] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for AirTime: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 19:03:39,076] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Late: str (schema_type=boolean feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 19:03:39,076] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Origin: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 19:03:39,076] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Dest: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-14 19:03:39,076] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Distance: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-14 19:03:39,076] [7842/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Late_avg: <class 'numpy.float64'> (schema_type=double feature_type=NUMERIC feature_role=TARGET)
[2021-04-14 19:03:39,076] [7842/MainThread] [INFO] [root] Reading with FIXED dtypes: {'dep_month': <class 'numpy.float64'>, 'dep_woy': <class 'numpy.float64'>, 'dep_dow': <class 'numpy.float64'>, 'dep_hour': <class 'numpy.float64'>, 'UniqueCarrier': 'str', 'FlightNum': <class 'numpy.float64'>, 'AirTime': <class 'numpy.float64'>, 'Late': 'str', 'Origin': 'str', 'Dest': 'str', 'Distance': <class 'numpy.float64'>, 'Late_avg': <class 'numpy.float64'>}
[2021-04-14 19:03:39,112] [7842/MainThread] [INFO] [root] Loaded table
[2021-04-14 19:03:39,113] [7842/MainThread] [INFO] [dataiku.doctor.prediction.background_rows_handler] Building background rows
[2021-04-14 19:03:39,114] [7842/MainThread] [INFO] [dataiku.doctor.prediction.background_rows_handler] Background estimated bias <= 0.0007567449862709852
[2021-04-14 19:03:39,118] [7842/MainThread] [INFO] [dataiku.doctor.prediction.background_rows_handler] Built background rows with shape=(1000, 12)
[2021-04-14 19:03:39,118] [7842/MainThread] [INFO] [dataiku.doctor.prediction.background_rows_handler] Saving background rows with shape=(1000, 12)
[2021-04-14 19:03:39,126] [7842/MainThread] [INFO] [dataiku.doctor.utils]  Coercion done
[2021-04-14 19:03:39,127] [7842/MainThread] [INFO] [dataiku.doctor.prediction.histogram_handler] Computing histograms
[2021-04-14 19:03:39,348] [7842/MainThread] [INFO] [dataiku.doctor.prediction.histogram_handler] Saving histograms
[2021-04-14 19:03:39,371] [7842/MainThread] [INFO] [dataiku.doctor.prediction.column_importance_handler] Computing column importance
/data/home/dkuacademy/Applications/dataiku-dss-9.0.0/python/dataiku/doctor/prediction/column_importance_handler.py:132: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples,), for example using ravel().
  clf.fit(preprocessed_array[idx], target[idx])
[2021-04-14 19:03:39,694] [7842/MainThread] [INFO] [dataiku.doctor.prediction.column_importance_handler] Saving column importance
[2021-04-14 19:03:39,699] [7842/MainThread] [INFO] [dataiku.base.socket_block_link] Client closed
[2021/04/14-19:03:39.699] [MRT-284011] [INFO] [dku.block.link.interaction]  - Check result for nullity exceptionIfNull=true result=not null
[2021/04/14-19:03:39.700] [MRT-284011] [INFO] [dku.analysis.prediction]  - Training returned ok
[2021/04/14-19:03:39.700] [MRT-284011] [INFO] [dku.ml.versioning]  - Dumping version info {"trainedWithDSSVersion":"9.0.0","trainedWithDSSConfVersion":"9000","trainedWithDSSBackend":"PY_MEMORY","backendCompatibilityVersion":18} in /data/home/dkuacademy/Library/dss-9.0/design/analysis-data/QS_DATAENG2/oSylVxXB/0IfwLWGQ/sessions/s7/pp1/m1
[2021/04/14-19:03:39.700] [MRT-284011] [INFO] [dku.analysis.ml]  - preTrain={
  "algorithm": "RIDGE_REGRESSION",
  "ridge_grid": {
    "alphaMode": "MANUAL",
    "alpha": {
      "gridMode": "EXPLICIT",
      "randomMode": "RANGE",
      "limit": {
        "min": 1.0E-23
      },
      "values": [
        0.1,
        1.0,
        3.0
      ],
      "range": {
        "min": 0.1,
        "max": 3.0,
        "nbValues": 3,
        "scaling": "LOGARITHMIC"
      }
    },
    "enabled": true
  },
  "max_ensemble_nodes_serialized": 0,
  "metrics": {
    "evaluationMetric": "R2",
    "customEvaluationMetricGIB": true,
    "customEvaluationMetricNeedsProba": false,
    "costMatrixWeights": {
      "tpGain": 1.0,
      "tnGain": 0.0,
      "fpGain": -0.3,
      "fnGain": 0.0
    },
    "liftPoint": 0.4
  },
  "autoOptimizeThreshold": true,
  "forcedClassifierThreshold": 0.0,
  "gridLength": 3,
  "grid_search_params": {
    "mode": "KFOLD",
    "splitRatio": 0.8,
    "shuffleIterations": 1,
    "nFolds": 5,
    "stratified": true,
    "strategy": "GRID",
    "bayesianOptimizer": "SCIKIT_OPTIMIZE",
    "randomized": true,
    "seed": 0,
    "nIter": 0,
    "nIterRandom": 3,
    "timeout": 0,
    "nJobs": 4,
    "distributed": false,
    "nContainers": 4
  },
  "pluginAlgoCustomGridSearch": false,
  "computeLearningCurves": false,
  "skipExpensiveReports": false
}
[2021/04/14-19:03:39.700] [MRT-284011] [INFO] [dku.analysis.ml]  - amp={
  "resolved": {
    "algorithm": "RIDGE_REGRESSION",
    "ridge": {
      "alpha": 0.1
    },
    "skipExpensiveReports": false
  },
  "other": {}
}
[2021/04/14-19:03:39.764] [MRT-284011] [WARN] [dku.kernels]  - Killing kernel python-single-command-kernel
[2021/04/14-19:03:39.764] [MRT-284011] [INFO] [dku.security.process]  - Killing process PID: 7842
[2021/04/14-19:03:39.767] [MRT-284011] [INFO] [dip.tickets]  - Destroying API ticket for analysis-ml-QS_DATAENG2-vdBizv6 on behalf of tara
[2021/04/14-19:03:39.793] [KNL-python-single-command-kernel-monitor-284014] [INFO] [dku.kernels]  - Process done with code 143
[2021/04/14-19:03:39.797] [KNL-python-single-command-kernel-out-284016] [DEBUG] [process]  - StreamToLine: EOF (stream closed)
[2021/04/14-19:03:39.797] [KNL-python-single-command-kernel-err-284017] [DEBUG] [process]  - StreamToLine: EOF (stream closed)
[2021/04/14-19:03:39.797] [KNL-python-single-command-kernel-monitor-284014] [WARN] [dku.resource]  - stat file for pid 7842 does not exist. Process died?
[2021/04/14-19:03:39.797] [KNL-python-single-command-kernel-monitor-284014] [INFO] [dku.resourceusage]  - Reporting completion of CRU:{"context":{"type":"ANALYSIS_ML_TRAIN","authIdentifier":"tara","projectKey":"QS_DATAENG2","analysisId":"oSylVxXB","mlTaskId":"0IfwLWGQ","sessionId":"s7"},"type":"LOCAL_PROCESS","id":"6YY9eDNH1XA1Qwng","startTime":1618419810341,"localProcess":{"pid":7842,"commandName":"/data/home/dkuacademy/Library/dss-9.0/design/code-envs/python/py_36_tara/bin/python","cpuUserTimeMS":9840,"cpuSystemTimeMS":8680,"cpuChildrenUserTimeMS":0,"cpuChildrenSystemTimeMS":0,"cpuTotalMS":18520,"cpuCurrent":3.704,"vmSizeMB":3549,"vmRSSMB":620,"vmHWMMB":1270,"vmRSSAnonMB":598,"vmDataMB":2859,"vmSizePeakMB":4131,"vmRSSPeakMB":620,"vmRSSTotalMBS":3100,"majorFaults":1,"childrenMajorFaults":0}}
[2021/04/14-19:03:39.798] [MRT-284011] [INFO] [dku.block.link]  - Closed socket
[2021/04/14-19:03:39.798] [MRT-284011] [INFO] [dku.block.link]  - Closed socket
[2021/04/14-19:03:39.798] [MRT-284011] [INFO] [dku.block.link]  - Closed serverSocket
[2021/04/14-19:03:39.798] [FT-TrainWorkThread-i4H0sQww-284009] [INFO] [dku.analysis.ml.python] T-0IfwLWGQ - Processing thread joined ...
[2021/04/14-19:03:39.798] [FT-TrainWorkThread-i4H0sQww-284009] [INFO] [dku.analysis.prediction] T-0IfwLWGQ - Train done
[2021/04/14-19:03:39.798] [FT-TrainWorkThread-i4H0sQww-284009] [INFO] [dku.analysis.prediction] T-0IfwLWGQ - Train done
[2021/04/14-19:03:39.800] [FT-TrainWorkThread-i4H0sQww-284009] [INFO] [dku.analysis.trainingdetails] T-0IfwLWGQ - Publishing mltask-train-done reflected event
