[2021/04/01-23:09:05.766] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.analysis.prediction]  - ******************************************
[2021/04/01-23:09:05.767] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.analysis.prediction]  - ** Start train session s1
[2021/04/01-23:09:05.767] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.analysis.prediction]  - ******************************************
[2021/04/01-23:09:05.769] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.shaker.data] T-0IfwLWGQ - [ct: 3] Need to compute sampleId before checking memory cache
[2021/04/01-23:09:05.769] [FT-TrainWorkThread-33cw3oHI-67773] [DEBUG] [dip.shaker.runner] T-0IfwLWGQ - [ct: 3] Script settings sampleMax=104857600 processedMax=-1
[2021/04/01-23:09:05.770] [FT-TrainWorkThread-33cw3oHI-67773] [DEBUG] [dip.shaker.runner] T-0IfwLWGQ - [ct: 4] Processing with sampleMax=104857600 processedMax=524288000
[2021/04/01-23:09:05.770] [FT-TrainWorkThread-33cw3oHI-67773] [DEBUG] [dip.shaker.runner] T-0IfwLWGQ - [ct: 4] Computed required sample id : b040f9a5a3c86a71de27ccdd36b31f09-NA-b9689059e8435a4efb7ef594c85e26650--d751713988987e9331980363e24189ce
[2021/04/01-23:09:05.770] [FT-TrainWorkThread-33cw3oHI-67773] [DEBUG] [dku.shaker.cache] T-0IfwLWGQ - Shaker MemoryCache get on dataset QSDE_SF_TO_FS_V2.flight_joined_prepared key=ds=165c3bad03d53a83d26a980b7f9c42ef--scr=df91f065784d2e0e7337ee17334938dc--samp=b040f9a5a3c86a71de27ccdd36b31f09-NA-b9689059e8435a4efb7ef594c85e26650--d751713988987e9331980363e24189ce: hit
[2021/04/01-23:09:05.771] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.shaker.schema] T-0IfwLWGQ - [ct: 5] Column dep_month meaning=LongMeaning fail=0
[2021/04/01-23:09:05.771] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.shaker.schema] T-0IfwLWGQ - [ct: 5] Column dep_woy meaning=LongMeaning fail=0
[2021/04/01-23:09:05.771] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.shaker.schema] T-0IfwLWGQ - [ct: 5] Column dep_dow meaning=LongMeaning fail=0
[2021/04/01-23:09:05.772] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.shaker.schema] T-0IfwLWGQ - [ct: 6] Column dep_hour meaning=LongMeaning fail=0
[2021/04/01-23:09:05.772] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.shaker.schema] T-0IfwLWGQ - [ct: 6] Column UniqueCarrier meaning=Text fail=0
[2021/04/01-23:09:05.772] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.shaker.schema] T-0IfwLWGQ - [ct: 6] Column FlightNum meaning=LongMeaning fail=0
[2021/04/01-23:09:05.772] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.shaker.schema] T-0IfwLWGQ - [ct: 6] Column AirTime meaning=LongMeaning fail=0
[2021/04/01-23:09:05.772] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.shaker.schema] T-0IfwLWGQ - [ct: 6] Column Late meaning=Boolean fail=0
[2021/04/01-23:09:05.772] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.shaker.schema] T-0IfwLWGQ - [ct: 6] Column Origin meaning=Text fail=0
[2021/04/01-23:09:05.772] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.shaker.schema] T-0IfwLWGQ - [ct: 6] Column Dest meaning=Text fail=0
[2021/04/01-23:09:05.772] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.shaker.schema] T-0IfwLWGQ - [ct: 6] Column Distance meaning=LongMeaning fail=0
[2021/04/01-23:09:05.773] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.shaker.schema] T-0IfwLWGQ - [ct: 7] Column Late_avg meaning=DoubleMeaning fail=0
[2021/04/01-23:09:05.778] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.datasets.file] T-0IfwLWGQ - [ct: 12] Building Filesystem handler config: {"connection":"filesystem_managed","path":"QSDE_SF_TO_FS_V2/flight_joined_prepared","notReadyIfEmpty":false,"filesSelectionRules":{"mode":"ALL","excludeRules":[],"includeRules":[],"explicitFiles":[]}}
[2021/04/01-23:09:05.779] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.datasets.ftplike] T-0IfwLWGQ - Enumerating Filesystem dataset prefix=
[2021/04/01-23:09:05.779] [FT-TrainWorkThread-33cw3oHI-67773] [DEBUG] [dku.fs.local] T-0IfwLWGQ - [ct: 13] Enumerating local filesystem prefix=/
[2021/04/01-23:09:05.779] [FT-TrainWorkThread-33cw3oHI-67773] [DEBUG] [dku.fs.local] T-0IfwLWGQ - [ct: 13] Enumeration done nb_paths=1 size=7831776
[2021/04/01-23:09:05.779] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.input.push] T-0IfwLWGQ - USTP: push selection.method=HEAD_SEQUENTIAL records=100000 ratio=0.02 col=null
[2021/04/01-23:09:05.780] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.format] T-0IfwLWGQ - [ct: 14] Extractor run: limit={"maxBytes":-1,"maxRecords":100000,"ordering":{"enabled":false,"rules":[]}} totalRecords=0
[2021/04/01-23:09:05.780] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku] T-0IfwLWGQ - getCompression filename=**out-s0.csv.gz**
[2021/04/01-23:09:05.780] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku] T-0IfwLWGQ - getCompression filename=**out-s0.csv.gz**
[2021/04/01-23:09:05.780] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.format] T-0IfwLWGQ - [ct: 14] Start compressed [GZIP] stream: /data/home/dkuacademy/Library/dss-9.0/design/managed_datasets/QSDE_SF_TO_FS_V2/flight_joined_prepared/out-s0.csv.gz / totalRecsBefore=0
[2021/04/01-23:09:05.780] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku] T-0IfwLWGQ - getCompression filename=**out-s0.csv.gz**
[2021/04/01-23:09:05.780] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku] T-0IfwLWGQ - getCompression filename=**out-s0.csv.gz**
[2021/04/01-23:09:06.611] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.format.csv] T-0IfwLWGQ - [ct: 845] CSV Emitted 100000 lines from file, 12 columns - interned: 100365 MEM: 96.1181640625%
[2021/04/01-23:09:06.611] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.format] T-0IfwLWGQ - [ct: 845] Limit reached
[2021/04/01-23:09:06.613] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 847] Checking if splits are up to date. Policy: type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337, instance id: 3c954e05fb55736ff7ecff41a6c5c14a-0
[2021/04/01-23:09:06.614] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 848] Search for split: p=type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337 i=3c954e05fb55736ff7ecff41a6c5c14a-0
[2021/04/01-23:09:06.615] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 849] Search for split: p=type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337 i=3c954e05fb55736ff7ecff41a6c5c14a-0
[2021/04/01-23:09:06.615] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 849] Checking if splits are up to date. Policy: type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337, instance id: 3c954e05fb55736ff7ecff41a6c5c14a-0
[2021/04/01-23:09:06.615] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 849] Search for split: p=type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337 i=3c954e05fb55736ff7ecff41a6c5c14a-0
[2021/04/01-23:09:06.616] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.analysis.splits] T-0IfwLWGQ - [ct: 850] Search for split: p=type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=flight_joined_prepared,sel=(method=head-s,records=100000),r=0.8,s=1337 i=3c954e05fb55736ff7ecff41a6c5c14a-0
[2021/04/01-23:09:06.618] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.analysis.ml.python] T-0IfwLWGQ - Joining processing thread ...
[2021/04/01-23:09:06.619] [MRT-67775] [INFO] [dku.analysis.ml.python]  - TrainAdditionalThread done
[2021/04/01-23:09:06.620] [MRT-67774] [INFO] [dku.analysis.ml.python]  - Running a preprocessing set: pp1 in /data/home/dkuacademy/Library/dss-9.0/design/analysis-data/QSDE_SF_TO_FS_V2/oSylVxXB/0IfwLWGQ/sessions/s1/pp1
[2021/04/01-23:09:06.621] [MRT-67774] [INFO] [dku.block.link]  - Started a socket on port 43369
[2021/04/01-23:09:06.634] [MRT-67774] [INFO] [dku.ml.kernel]  - Writing output of python-single-command-kernel to /data/home/dkuacademy/Library/dss-9.0/design/analysis-data/QSDE_SF_TO_FS_V2/oSylVxXB/0IfwLWGQ/sessions/s1/pp1/train.log
[2021/04/01-23:09:06.640] [MRT-67774] [INFO] [dku.code.envs.resolution]  - Executing Python activity in builtin env
[2021/04/01-23:09:06.640] [MRT-67774] [WARN] [dku.code.projectLibs]  - External libraries file not found: /data/home/dkuacademy/Library/dss-9.0/design/config/projects/QSDE_SF_TO_FS_V2/lib/external-libraries.json
[2021/04/01-23:09:06.640] [MRT-67774] [INFO] [dku.code.projectLibs]  - EXTERNAL LIBS FROM QSDE_SF_TO_FS_V2 is {"gitReferences":{},"pythonPath":["python"],"rsrcPath":["R"],"importLibrariesFromProjects":[]}
[2021/04/01-23:09:06.641] [MRT-67774] [INFO] [dku.code.projectLibs]  - chunkFolder is /data/home/dkuacademy/Library/dss-9.0/design/config/projects/QSDE_SF_TO_FS_V2/lib/R
[2021/04/01-23:09:06.641] [MRT-67774] [INFO] [dku.python.single_command.kernel]  - Starting Python process for kernel  python-single-command-kernel
[2021/04/01-23:09:06.641] [MRT-67774] [INFO] [dip.tickets]  - Creating API ticket for analysis-ml-QSDE_SF_TO_FS_V2-DEz4pzP on behalf of tara id=analysis-ml-QSDE_SF_TO_FS_V2-DEz4pzP_zB0aLYpyyzBW
[2021/04/01-23:09:06.650] [MRT-67774] [INFO] [dku.security.process]  - Starting process (regular)
[2021/04/01-23:09:06.652] [MRT-67774] [INFO] [dku.security.process]  - Process started with pid=4003
[2021/04/01-23:09:06.652] [MRT-67774] [INFO] [dku.processes.cgroups]  - Will use cgroups []
[2021/04/01-23:09:06.652] [MRT-67774] [INFO] [dku.processes.cgroups]  - Applying rules to used cgroups: []
[2021/04/01-23:09:06.653] [KNL-python-single-command-kernel-monitor-67779] [INFO] [dku.resourceusage]  - Reporting start of CRU:{"context":{"type":"ANALYSIS_ML_TRAIN","authIdentifier":"tara","projectKey":"QSDE_SF_TO_FS_V2","analysisId":"oSylVxXB","mlTaskId":"0IfwLWGQ","sessionId":"s1"},"type":"LOCAL_PROCESS","id":"vi4lq9dOAyaEglTq","startTime":1617311346653,"localProcess":{"cpuCurrent":0.0}}
[2021/04/01-23:09:06.655] [process-resource-monitor-4003-67780] [DEBUG] [dku.resource]  - Process stats for pid 4003: {"pid":4003,"commandName":"/data/home/dkuacademy/Library/dss-9.0/design/bin/python","cpuUserTimeMS":0,"cpuSystemTimeMS":0,"cpuChildrenUserTimeMS":0,"cpuChildrenSystemTimeMS":0,"cpuTotalMS":0,"cpuCurrent":0.0,"vmSizeMB":18,"vmRSSMB":2,"vmHWMMB":2,"vmRSSAnonMB":1,"vmDataMB":1,"vmSizePeakMB":18,"vmRSSPeakMB":2,"vmRSSTotalMBS":0,"majorFaults":0,"childrenMajorFaults":0}
Installing debugging signal handler
[2021-04-01 23:09:07,147] [4003/MainThread] [INFO] [dataiku.base.socket_block_link] Connecting to localhost (127.0.0.1) at port 43369
[2021-04-01 23:09:07,147] [4003/MainThread] [INFO] [dataiku.base.socket_block_link] Connected to localhost (127.0.0.1) at port 43369
[2021/04/01-23:09:07.148] [MRT-67774] [INFO] [dku.link.secret_protected]  - Connected to kernel
[2021/04/01-23:09:07.160] [MRT-67774] [INFO] [dku.block.link.interaction]  - Execute link command respClazz=true respTypeToken=false respIsString=false is=false asyncInputStream=false os=false
[2021-04-01 23:09:07,896] [4003/MainThread] [INFO] [dataiku.doctor.utils.dku_pickle] Setting cloudpickle as the pickling tool
[2021-04-01 23:09:07,954] [4003/MainThread] [INFO] [root] Running analysis command: train_prediction_models_nosave
[2021-04-01 23:09:07,955] [4003/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: DatasetSanityCheckDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_DATASET_SANITY_CHECKS
[2021-04-01 23:09:07,955] [4003/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: ClassifierAccuracyCheckDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_MODEL_CHECK
[2021-04-01 23:09:07,955] [4003/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: RegressionR2CheckDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_MODEL_CHECK
[2021-04-01 23:09:07,955] [4003/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: LeakageDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_LEAKAGE_DETECTION
[2021-04-01 23:09:07,955] [4003/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: TreeOverfitDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_TRAINING_OVERFIT
[2021-04-01 23:09:07,955] [4003/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: MLAssertionsDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_ML_ASSERTIONS
[2021-04-01 23:09:07,955] [4003/MainThread] [INFO] [dataiku.doctor.commands] PPS is {u'preprocessingFitSampleSeed': 1337, u'feature_selection_params': {u'custom_params': {u'code': u'# type your code here'}, u'pca_params': {u'variance_proportion': 0.9, u'n_features': 25}, u'random_forest_params': {u'depth': 10, u'n_features': 25, u'n_trees': 30}, u'lasso_params': {u'alpha': [0.01, 0.1, 1.0, 10.0, 100.0], u'cross_validate': True}, u'method': u'NONE', u'correlation_params': {u'max_abs_correlation': 1.0, u'n_features': 25, u'min_abs_correlation': 0.0}}, u'preprocessingFitSampleRatio': 1.0, u'reduce': {u'enabled': False, u'kept_variance': 0.0}, u'skipPreprocessing': False, u'target_remapping': [], u'per_feature': {u'Origin': {u'missing_impute_with': u'MODE', u'autoReason': u'REJECT_CARDINALITY', u'hash_whole_categories': True, u'nb_bins_hashing': 1048576, u'customHandlingCode': u'', u'category_handling': u'DUMMIFY', u'cumulative_proportion': 0.95, u'sendToInput': u'main', u'state': {u'userModified': False, u'recordedMeaning': u'Text', u'autoModifiedByDSS': False}, u'missing_handling': u'NONE', u'max_nb_categories': 100, u'customProcessorWantsMatrix': False, u'dummy_clip': u'MAX_NB_CATEGORIES', u'role': u'REJECT', u'max_cat_safety': 200, u'min_samples': 10, u'type': u'CATEGORY', u'dummy_drop': u'NONE'}, u'dep_hour': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'UniqueCarrier': {u'missing_impute_with': u'MODE', u'customProcessorWantsMatrix': False, u'hash_whole_categories': True, u'nb_bins_hashing': 1048576, u'customHandlingCode': u'', u'category_handling': u'DUMMIFY', u'cumulative_proportion': 0.95, u'sendToInput': u'main', u'state': {u'userModified': False, u'recordedMeaning': u'Text', u'autoModifiedByDSS': False}, u'missing_handling': u'NONE', u'max_nb_categories': 5, u'dummy_clip': u'MAX_NB_CATEGORIES', u'role': u'INPUT', u'max_cat_safety': 200, u'min_samples': 10, u'type': u'CATEGORY', u'dummy_drop': u'NONE'}, u'FlightNum': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'Late_avg': {u'generate_derivative': False, u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'sendToInput': u'main', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'DoubleMeaning', u'autoModifiedByDSS': False}, u'role': u'TARGET', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'dep_woy': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'Distance': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'Dest': {u'missing_impute_with': u'MODE', u'autoReason': u'REJECT_CARDINALITY', u'hash_whole_categories': True, u'nb_bins_hashing': 1048576, u'customHandlingCode': u'', u'category_handling': u'DUMMIFY', u'cumulative_proportion': 0.95, u'sendToInput': u'main', u'state': {u'userModified': False, u'recordedMeaning': u'Text', u'autoModifiedByDSS': False}, u'missing_handling': u'NONE', u'max_nb_categories': 100, u'customProcessorWantsMatrix': False, u'dummy_clip': u'MAX_NB_CATEGORIES', u'role': u'REJECT', u'max_cat_safety': 200, u'min_samples': 10, u'type': u'CATEGORY', u'dummy_drop': u'NONE'}, u'dep_month': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'dep_dow': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'Late': {u'missing_impute_with': u'MODE', u'customProcessorWantsMatrix': False, u'hash_whole_categories': True, u'nb_bins_hashing': 1048576, u'customHandlingCode': u'', u'category_handling': u'DUMMIFY', u'cumulative_proportion': 0.95, u'sendToInput': u'main', u'state': {u'userModified': False, u'recordedMeaning': u'Boolean', u'autoModifiedByDSS': False}, u'missing_handling': u'NONE', u'max_nb_categories': 5, u'dummy_clip': u'MAX_NB_CATEGORIES', u'role': u'INPUT', u'max_cat_safety': 200, u'min_samples': 10, u'type': u'CATEGORY', u'dummy_drop': u'NONE'}, u'AirTime': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}}, u'feature_generation': {u'manual_interactions': {u'interactions': []}, u'pairwise_linear': {u'behavior': u'DISABLED'}, u'categoricals_count_transformer': {u'input_features': [], u'all_features': False, u'behavior': u'DISABLED'}, u'polynomial_combinations': {u'behavior': u'DISABLED'}, u'numericals_clustering': {u'k': 0, u'input_features': [], u'all_features': False, u'behavior': u'DISABLED'}}}
[2021-04-01 23:09:07,955] [4003/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Loading train set
[2021-04-01 23:09:07,992] [4003/MainThread] [INFO] [root] Reading with dtypes: None
[2021-04-01 23:09:07,992] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_month: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-01 23:09:07,992] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_woy: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-01 23:09:07,992] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_dow: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-01 23:09:07,992] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_hour: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-01 23:09:07,992] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for UniqueCarrier: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-01 23:09:07,992] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for FlightNum: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-01 23:09:07,992] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for AirTime: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-01 23:09:07,992] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Late: str (schema_type=boolean feature_type=CATEGORY feature_role=INPUT)
[2021-04-01 23:09:07,992] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Origin: None (schema_type=string feature_type=CATEGORY feature_role=REJECT)
[2021-04-01 23:09:07,992] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Dest: None (schema_type=string feature_type=CATEGORY feature_role=REJECT)
[2021-04-01 23:09:07,992] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Distance: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-01 23:09:07,992] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Late_avg: <type 'numpy.float64'> (schema_type=double feature_type=NUMERIC feature_role=TARGET)
[2021-04-01 23:09:07,992] [4003/MainThread] [INFO] [root] Reading with FIXED dtypes: {u'dep_hour': <type 'numpy.float64'>, u'Late': 'str', u'UniqueCarrier': 'str', u'FlightNum': <type 'numpy.float64'>, u'Late_avg': <type 'numpy.float64'>, u'dep_woy': <type 'numpy.float64'>, u'Distance': <type 'numpy.float64'>, u'dep_month': <type 'numpy.float64'>, u'dep_dow': <type 'numpy.float64'>, u'AirTime': <type 'numpy.float64'>}
[2021-04-01 23:09:08,105] [4003/MainThread] [INFO] [root] Loaded table
[2021-04-01 23:09:08,105] [4003/MainThread] [INFO] [dataiku.doctor.utils]  Coercion done
[2021-04-01 23:09:08,105] [4003/MainThread] [INFO] [dataiku.doctor.utils.split] Loaded train df: shape=(79912,12)
[2021-04-01 23:09:08,105] [4003/MainThread] [INFO] [dataiku.doctor.commands] Train col : dep_month (float64)
[2021-04-01 23:09:08,106] [4003/MainThread] [INFO] [dataiku.doctor.commands] Train col : dep_woy (float64)
[2021-04-01 23:09:08,106] [4003/MainThread] [INFO] [dataiku.doctor.commands] Train col : dep_dow (float64)
[2021-04-01 23:09:08,106] [4003/MainThread] [INFO] [dataiku.doctor.commands] Train col : dep_hour (float64)
[2021-04-01 23:09:08,106] [4003/MainThread] [INFO] [dataiku.doctor.commands] Train col : UniqueCarrier (object)
[2021-04-01 23:09:08,106] [4003/MainThread] [INFO] [dataiku.doctor.commands] Train col : FlightNum (float64)
[2021-04-01 23:09:08,106] [4003/MainThread] [INFO] [dataiku.doctor.commands] Train col : AirTime (float64)
[2021-04-01 23:09:08,106] [4003/MainThread] [INFO] [dataiku.doctor.commands] Train col : Late (object)
[2021-04-01 23:09:08,106] [4003/MainThread] [INFO] [dataiku.doctor.commands] Train col : Origin (object)
[2021-04-01 23:09:08,106] [4003/MainThread] [INFO] [dataiku.doctor.commands] Train col : Dest (object)
[2021-04-01 23:09:08,106] [4003/MainThread] [INFO] [dataiku.doctor.commands] Train col : Distance (float64)
[2021-04-01 23:09:08,106] [4003/MainThread] [INFO] [dataiku.doctor.commands] Train col : Late_avg (float64)
[2021-04-01 23:09:08,106] [4003/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Loading train set
[2021-04-01 23:09:08,106] [4003/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Loading test set
[2021-04-01 23:09:08,106] [4003/MainThread] [INFO] [root] Reading with dtypes: None
[2021-04-01 23:09:08,106] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_month: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-01 23:09:08,107] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_woy: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-01 23:09:08,107] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_dow: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-01 23:09:08,107] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_hour: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-01 23:09:08,107] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for UniqueCarrier: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-01 23:09:08,107] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for FlightNum: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-01 23:09:08,107] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for AirTime: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-01 23:09:08,107] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Late: str (schema_type=boolean feature_type=CATEGORY feature_role=INPUT)
[2021-04-01 23:09:08,107] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Origin: None (schema_type=string feature_type=CATEGORY feature_role=REJECT)
[2021-04-01 23:09:08,107] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Dest: None (schema_type=string feature_type=CATEGORY feature_role=REJECT)
[2021-04-01 23:09:08,107] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Distance: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-01 23:09:08,107] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Late_avg: <type 'numpy.float64'> (schema_type=double feature_type=NUMERIC feature_role=TARGET)
[2021-04-01 23:09:08,107] [4003/MainThread] [INFO] [root] Reading with FIXED dtypes: {u'dep_hour': <type 'numpy.float64'>, u'Late': 'str', u'UniqueCarrier': 'str', u'FlightNum': <type 'numpy.float64'>, u'Late_avg': <type 'numpy.float64'>, u'dep_woy': <type 'numpy.float64'>, u'Distance': <type 'numpy.float64'>, u'dep_month': <type 'numpy.float64'>, u'dep_dow': <type 'numpy.float64'>, u'AirTime': <type 'numpy.float64'>}
[2021-04-01 23:09:08,144] [4003/MainThread] [INFO] [root] Loaded table
[2021-04-01 23:09:08,145] [4003/MainThread] [INFO] [dataiku.doctor.utils]  Coercion done
[2021-04-01 23:09:08,145] [4003/MainThread] [INFO] [dataiku.doctor.utils.split] Loaded test df: shape=(20088,12)
[2021-04-01 23:09:08,145] [4003/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Loading test set
[2021-04-01 23:09:08,145] [4003/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Collecting statistics
[2021-04-01 23:09:08,146] [4003/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Origin... (type=CATEGORY)
[2021-04-01 23:09:08,146] [4003/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at dep_hour... (type=NUMERIC)
[2021-04-01 23:09:08,146] [4003/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-01 23:09:08,149] [4003/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at UniqueCarrier... (type=CATEGORY)
[2021-04-01 23:09:08,161] [4003/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at FlightNum... (type=NUMERIC)
[2021-04-01 23:09:08,161] [4003/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-01 23:09:08,164] [4003/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Late_avg... (type=NUMERIC)
[2021-04-01 23:09:08,164] [4003/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at dep_woy... (type=NUMERIC)
[2021-04-01 23:09:08,164] [4003/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-01 23:09:08,166] [4003/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Distance... (type=NUMERIC)
[2021-04-01 23:09:08,166] [4003/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-01 23:09:08,169] [4003/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Dest... (type=CATEGORY)
[2021-04-01 23:09:08,169] [4003/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at dep_month... (type=NUMERIC)
[2021-04-01 23:09:08,169] [4003/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-01 23:09:08,171] [4003/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at dep_dow... (type=NUMERIC)
[2021-04-01 23:09:08,171] [4003/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-01 23:09:08,174] [4003/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Late... (type=CATEGORY)
[2021-04-01 23:09:08,186] [4003/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at AirTime... (type=NUMERIC)
[2021-04-01 23:09:08,186] [4003/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2021-04-01 23:09:08,189] [4003/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Collecting statistics
[2021-04-01 23:09:08,189] [4003/MainThread] [INFO] [dataiku.doctor.multiframe] generating interactions
[2021-04-01 23:09:08,189] [4003/MainThread] [INFO] [dataiku.doctor.multiframe] {u'preprocessingFitSampleSeed': 1337, u'feature_selection_params': {u'custom_params': {u'code': u'# type your code here'}, u'pca_params': {u'variance_proportion': 0.9, u'n_features': 25}, u'random_forest_params': {u'depth': 10, u'n_features': 25, u'n_trees': 30}, u'lasso_params': {u'alpha': [0.01, 0.1, 1.0, 10.0, 100.0], u'cross_validate': True}, u'method': u'NONE', u'correlation_params': {u'max_abs_correlation': 1.0, u'n_features': 25, u'min_abs_correlation': 0.0}}, u'preprocessingFitSampleRatio': 1.0, u'reduce': {u'enabled': False, u'kept_variance': 0.0}, u'skipPreprocessing': False, u'target_remapping': [], u'per_feature': {u'Origin': {u'missing_impute_with': u'MODE', u'autoReason': u'REJECT_CARDINALITY', u'hash_whole_categories': True, u'nb_bins_hashing': 1048576, u'customHandlingCode': u'', u'category_handling': u'DUMMIFY', u'cumulative_proportion': 0.95, u'sendToInput': u'main', u'state': {u'userModified': False, u'recordedMeaning': u'Text', u'autoModifiedByDSS': False}, u'missing_handling': u'NONE', u'max_nb_categories': 100, u'customProcessorWantsMatrix': False, u'dummy_clip': u'MAX_NB_CATEGORIES', u'role': u'REJECT', u'max_cat_safety': 200, u'min_samples': 10, u'type': u'CATEGORY', u'dummy_drop': u'NONE'}, u'dep_hour': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'UniqueCarrier': {u'missing_impute_with': u'MODE', u'customProcessorWantsMatrix': False, u'hash_whole_categories': True, u'nb_bins_hashing': 1048576, u'customHandlingCode': u'', u'category_handling': u'DUMMIFY', u'cumulative_proportion': 0.95, u'sendToInput': u'main', u'state': {u'userModified': False, u'recordedMeaning': u'Text', u'autoModifiedByDSS': False}, u'missing_handling': u'NONE', u'max_nb_categories': 5, u'dummy_clip': u'MAX_NB_CATEGORIES', u'role': u'INPUT', u'max_cat_safety': 200, u'min_samples': 10, u'type': u'CATEGORY', u'dummy_drop': u'NONE'}, u'FlightNum': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'Late_avg': {u'generate_derivative': False, u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'sendToInput': u'main', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'DoubleMeaning', u'autoModifiedByDSS': False}, u'role': u'TARGET', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'dep_woy': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'Distance': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'Dest': {u'missing_impute_with': u'MODE', u'autoReason': u'REJECT_CARDINALITY', u'hash_whole_categories': True, u'nb_bins_hashing': 1048576, u'customHandlingCode': u'', u'category_handling': u'DUMMIFY', u'cumulative_proportion': 0.95, u'sendToInput': u'main', u'state': {u'userModified': False, u'recordedMeaning': u'Text', u'autoModifiedByDSS': False}, u'missing_handling': u'NONE', u'max_nb_categories': 100, u'customProcessorWantsMatrix': False, u'dummy_clip': u'MAX_NB_CATEGORIES', u'role': u'REJECT', u'max_cat_safety': 200, u'min_samples': 10, u'type': u'CATEGORY', u'dummy_drop': u'NONE'}, u'dep_month': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'dep_dow': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}, u'Late': {u'missing_impute_with': u'MODE', u'customProcessorWantsMatrix': False, u'hash_whole_categories': True, u'nb_bins_hashing': 1048576, u'customHandlingCode': u'', u'category_handling': u'DUMMIFY', u'cumulative_proportion': 0.95, u'sendToInput': u'main', u'state': {u'userModified': False, u'recordedMeaning': u'Boolean', u'autoModifiedByDSS': False}, u'missing_handling': u'NONE', u'max_nb_categories': 5, u'dummy_clip': u'MAX_NB_CATEGORIES', u'role': u'INPUT', u'max_cat_safety': 200, u'min_samples': 10, u'type': u'CATEGORY', u'dummy_drop': u'NONE'}, u'AirTime': {u'generate_derivative': False, u'sendToInput': u'main', u'rescaling': u'AVGSTD', u'role': u'INPUT', u'customHandlingCode': u'', u'customProcessorWantsMatrix': False, u'numerical_handling': u'REGULAR', u'binarize_threshold_mode': u'MEDIAN', u'state': {u'userModified': False, u'recordedMeaning': u'LongMeaning', u'autoModifiedByDSS': False}, u'missing_handling': u'IMPUTE', u'binarize_constant_threshold': 0.0, u'quantile_bin_nb_bins': 4, u'missing_impute_with': u'MEAN', u'type': u'NUMERIC', u'impute_constant_value': 0.0}}, u'feature_generation': {u'manual_interactions': {u'interactions': []}, u'pairwise_linear': {u'behavior': u'DISABLED'}, u'categoricals_count_transformer': {u'input_features': [], u'all_features': False, u'behavior': u'DISABLED'}, u'polynomial_combinations': {u'behavior': u'DISABLED'}, u'numericals_clustering': {u'k': 0, u'input_features': [], u'all_features': False, u'behavior': u'DISABLED'}}}
[2021-04-01 23:09:08,189] [4003/MainThread] [INFO] [dataiku.doctor.multiframe] No feature selection to perform
[2021-04-01 23:09:08,189] [4003/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Preprocessing train set
[2021-04-01 23:09:08,190] [4003/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 79912
[2021-04-01 23:09:08,190] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RemapValueToOutput
[2021-04-01 23:09:08,190] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:MultipleImputeMissingFromInput
[2021-04-01 23:09:08,190] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {u'dep_hour': 13.086732906196817, u'Distance': 724.2432175392933, u'FlightNum': 2149.660088597457, u'dep_woy': 14.544386324957452, u'dep_month': 3.7664931424567025, u'dep_dow': 3.9422364601061166, u'AirTime': 103.37577585343878}
[2021-04-01 23:09:08,194] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (dep_hour)
[2021-04-01 23:09:08,195] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_hour (avg=13.0867329062 std=4.80188247322 shift=13.0867329062 inv_scale=0.208251660797)
[2021-04-01 23:09:08,211] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_hour (avg=6.69312549233e-16 std=1.0) nulls=0
[2021-04-01 23:09:08,211] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (FlightNum)
[2021-04-01 23:09:08,212] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale FlightNum (avg=2149.6600886 std=1944.09274632 shift=2149.6600886 inv_scale=0.000514378751678)
[2021-04-01 23:09:08,214] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled FlightNum (avg=3.87140820508e-16 std=1.0) nulls=0
[2021-04-01 23:09:08,214] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (dep_woy)
[2021-04-01 23:09:08,215] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_woy (avg=14.544386325 std=6.97121381815 shift=14.544386325 inv_scale=0.143447041804)
[2021-04-01 23:09:08,217] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_woy (avg=2.01557300121e-13 std=1.0) nulls=0
[2021-04-01 23:09:08,217] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (Distance)
[2021-04-01 23:09:08,218] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale Distance (avg=724.243217539 std=561.810411687 shift=724.243217539 inv_scale=0.00177995989252)
[2021-04-01 23:09:08,220] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled Distance (avg=1.16862949168e-16 std=1.0) nulls=0
[2021-04-01 23:09:08,220] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (dep_month)
[2021-04-01 23:09:08,220] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_month (avg=3.76649314246 std=1.59191042442 shift=3.76649314246 inv_scale=0.628176048512)
[2021-04-01 23:09:08,223] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_month (avg=1.15324419462e-13 std=1.0) nulls=0
[2021-04-01 23:09:08,223] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (dep_dow)
[2021-04-01 23:09:08,223] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_dow (avg=3.94223646011 std=1.9826725071 shift=3.94223646011 inv_scale=0.50436973147)
[2021-04-01 23:09:08,225] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_dow (avg=1.69635498405e-15 std=1.0) nulls=0
[2021-04-01 23:09:08,225] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (AirTime)
[2021-04-01 23:09:08,226] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale AirTime (avg=103.375775853 std=68.4857135559 shift=103.375775853 inv_scale=0.0146015854706)
[2021-04-01 23:09:08,228] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled AirTime (avg=5.50103338372e-15 std=1.0) nulls=0
[2021-04-01 23:09:08,228] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FlushDFBuilder(num_flagonly)
[2021-04-01 23:09:08,228] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FastSparseDummifyProcessor (UniqueCarrier)
[2021-04-01 23:09:08,244] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(79912, 8) nnz=79912
[2021-04-01 23:09:08,244] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FastSparseDummifyProcessor (Late)
[2021-04-01 23:09:08,259] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(79912, 4) nnz=79912
[2021-04-01 23:09:08,260] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:MultipleImputeMissingFromInput
[2021-04-01 23:09:08,260] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2021-04-01 23:09:08,260] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FlushDFBuilder(cat_flagpresence)
[2021-04-01 23:09:08,260] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:MultipleImputeMissingFromInput
[2021-04-01 23:09:08,260] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2021-04-01 23:09:08,260] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FlushDFBuilder(interaction)
[2021-04-01 23:09:08,260] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RealignTarget
[2021-04-01 23:09:08,260] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] Realign target series = (79912,)
[2021-04-01 23:09:08,262] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] After realign target: (79912,)
[2021-04-01 23:09:08,262] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:DropRowsWhereNoTarget
[2021-04-01 23:09:08,263] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] Deleting 0 rows because one of ['target'] is missing
[2021-04-01 23:09:08,263] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] MF before = (79912, 19)
[2021-04-01 23:09:08,264] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] target before = (79912,)
[2021-04-01 23:09:08,268] [4003/MainThread] [INFO] [dataiku.doctor.multiframe] MultiFrame, dropping rows: []
[2021-04-01 23:09:08,289] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] After DRWNT input_df=(79912, 12)
[2021-04-01 23:09:08,289] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] MF after = (79912, 19)
[2021-04-01 23:09:08,289] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] target after = (79912,)
[2021-04-01 23:09:08,289] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:DumpPipelineState
[2021-04-01 23:09:08,289] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] ********* Pipeline state (Before feature selection)
[2021-04-01 23:09:08,289] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]    input_df= (79912, 12) 
[2021-04-01 23:09:08,289] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]    current_mf=(79912, 19) 
[2021-04-01 23:09:08,289] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]    PPR: 
[2021-04-01 23:09:08,289] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]       target = <class 'pandas.core.series.Series'> ((79912,))
[2021-04-01 23:09:08,289] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:EmitCurrentMFAsResult
[2021-04-01 23:09:08,289] [4003/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 79912
[2021-04-01 23:09:08,289] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:DumpPipelineState
[2021-04-01 23:09:08,289] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] ********* Pipeline state (At end)
[2021-04-01 23:09:08,289] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]    input_df= (79912, 12) 
[2021-04-01 23:09:08,289] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]    current_mf=(0, 0) 
[2021-04-01 23:09:08,289] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]    PPR: 
[2021-04-01 23:09:08,289] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]       UNPROCESSED = <class 'pandas.core.frame.DataFrame'> ((79912, 12))
[2021-04-01 23:09:08,290] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]       TRAIN = <class 'dataiku.doctor.multiframe.MultiFrame'> ((79912, 19))
[2021-04-01 23:09:08,290] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]       target = <class 'pandas.core.series.Series'> ((79912,))
[2021-04-01 23:09:08,291] [4003/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Preprocessing train set
[2021-04-01 23:09:08,291] [4003/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Preprocessing test set
[2021-04-01 23:09:08,291] [4003/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 20088
[2021-04-01 23:09:08,291] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RemapValueToOutput
[2021-04-01 23:09:08,292] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:MultipleImputeMissingFromInput
[2021-04-01 23:09:08,292] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {u'dep_hour': 13.086732906196817, u'Distance': 724.2432175392933, u'FlightNum': 2149.660088597457, u'dep_woy': 14.544386324957452, u'dep_month': 3.7664931424567025, u'dep_dow': 3.9422364601061166, u'AirTime': 103.37577585343878}
[2021-04-01 23:09:08,293] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (dep_hour)
[2021-04-01 23:09:08,294] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_hour (avg=13.1375448029 std=4.81152184405 shift=13.0867329062 inv_scale=0.208251660797)
[2021-04-01 23:09:08,295] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_hour (avg=0.0105816618699 std=1.00200741499) nulls=0
[2021-04-01 23:09:08,295] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (FlightNum)
[2021-04-01 23:09:08,295] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale FlightNum (avg=2129.26144962 std=1938.80005733 shift=2149.6600886 inv_scale=0.000514378751678)
[2021-04-01 23:09:08,297] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled FlightNum (avg=-0.0104926264523 std=0.997277553245) nulls=0
[2021-04-01 23:09:08,297] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (dep_woy)
[2021-04-01 23:09:08,297] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_woy (avg=14.5089107925 std=6.93786261177 shift=14.544386325 inv_scale=0.143447041804)
[2021-04-01 23:09:08,298] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_woy (avg=-0.00508886018562 std=0.995215868103) nulls=0
[2021-04-01 23:09:08,298] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (Distance)
[2021-04-01 23:09:08,298] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale Distance (avg=726.248058542 std=562.197650727 shift=724.243217539 inv_scale=0.00177995989252)
[2021-04-01 23:09:08,300] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled Distance (avg=0.00356853657642 std=1.00068926996) nulls=0
[2021-04-01 23:09:08,300] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (dep_month)
[2021-04-01 23:09:08,300] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_month (avg=3.75985663082 std=1.58189507051 shift=3.76649314246 inv_scale=0.628176048512)
[2021-04-01 23:09:08,301] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_month (avg=-0.00416889765302 std=0.993708594551) nulls=0
[2021-04-01 23:09:08,302] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (dep_dow)
[2021-04-01 23:09:08,302] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale dep_dow (avg=3.92751891677 std=1.98353439579 shift=3.94223646011 inv_scale=0.50436973147)
[2021-04-01 23:09:08,303] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled dep_dow (avg=-0.00742308338224 std=1.00043471057) nulls=0
[2021-04-01 23:09:08,303] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (AirTime)
[2021-04-01 23:09:08,303] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale AirTime (avg=103.513590203 std=68.678091524 shift=103.375775853 inv_scale=0.0146015854706)
[2021-04-01 23:09:08,305] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled AirTime (avg=0.00201230800574 std=1.00280902334) nulls=0
[2021-04-01 23:09:08,305] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FlushDFBuilder(num_flagonly)
[2021-04-01 23:09:08,305] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FastSparseDummifyProcessor (UniqueCarrier)
[2021-04-01 23:09:08,309] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(20088, 8) nnz=20088
[2021-04-01 23:09:08,310] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FastSparseDummifyProcessor (Late)
[2021-04-01 23:09:08,314] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(20088, 4) nnz=20088
[2021-04-01 23:09:08,314] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:MultipleImputeMissingFromInput
[2021-04-01 23:09:08,314] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2021-04-01 23:09:08,314] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FlushDFBuilder(cat_flagpresence)
[2021-04-01 23:09:08,314] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:MultipleImputeMissingFromInput
[2021-04-01 23:09:08,314] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2021-04-01 23:09:08,314] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FlushDFBuilder(interaction)
[2021-04-01 23:09:08,314] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RealignTarget
[2021-04-01 23:09:08,314] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] Realign target series = (20088,)
[2021-04-01 23:09:08,315] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] After realign target: (20088,)
[2021-04-01 23:09:08,315] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:DropRowsWhereNoTarget
[2021-04-01 23:09:08,315] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] Deleting 0 rows because one of ['target'] is missing
[2021-04-01 23:09:08,315] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] MF before = (20088, 19)
[2021-04-01 23:09:08,315] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] target before = (20088,)
[2021-04-01 23:09:08,316] [4003/MainThread] [INFO] [dataiku.doctor.multiframe] MultiFrame, dropping rows: []
[2021-04-01 23:09:08,322] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] After DRWNT input_df=(20088, 12)
[2021-04-01 23:09:08,322] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] MF after = (20088, 19)
[2021-04-01 23:09:08,322] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] target after = (20088,)
[2021-04-01 23:09:08,322] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:DumpPipelineState
[2021-04-01 23:09:08,322] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] ********* Pipeline state (Before feature selection)
[2021-04-01 23:09:08,322] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]    input_df= (20088, 12) 
[2021-04-01 23:09:08,322] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]    current_mf=(20088, 19) 
[2021-04-01 23:09:08,322] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]    PPR: 
[2021-04-01 23:09:08,322] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]       target = <class 'pandas.core.series.Series'> ((20088,))
[2021-04-01 23:09:08,322] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:EmitCurrentMFAsResult
[2021-04-01 23:09:08,322] [4003/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 20088
[2021-04-01 23:09:08,322] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:DumpPipelineState
[2021-04-01 23:09:08,322] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] ********* Pipeline state (At end)
[2021-04-01 23:09:08,322] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]    input_df= (20088, 12) 
[2021-04-01 23:09:08,322] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]    current_mf=(0, 0) 
[2021-04-01 23:09:08,322] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]    PPR: 
[2021-04-01 23:09:08,322] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]       UNPROCESSED = <class 'pandas.core.frame.DataFrame'> ((20088, 12))
[2021-04-01 23:09:08,322] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]       TRAIN = <class 'dataiku.doctor.multiframe.MultiFrame'> ((20088, 19))
[2021-04-01 23:09:08,322] [4003/MainThread] [DEBUG] [dku.ml.preprocessing]       target = <class 'pandas.core.series.Series'> ((20088,))
[2021-04-01 23:09:08,322] [4003/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Preprocessing test set
[2021-04-01 23:09:08,323] [4003/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Fitting model
[2021-04-01 23:09:08,331] [4003/MainThread] [DEBUG] [dku.ml.preprocessing] Deleting 0 rows
[2021-04-01 23:09:08,332] [4003/MainThread] [INFO] [dataiku.doctor.multiframe] MultiFrame, dropping rows: []
[2021-04-01 23:09:08,342] [4003/MainThread] [INFO] [dataiku.doctor.prediction.common] prepare multiframe shape=(79912,19) tn=1518328 nnz=719208 fill_ratio=0.47
[2021-04-01 23:09:08,342] [4003/MainThread] [INFO] [dataiku.doctor.prediction.common] too small, using array
[2021-04-01 23:09:08,349] [4003/MainThread] [INFO] [dataiku.doctor.prediction.common] Create CLF from params: {u'computeLearningCurves': False, u'rf_regressor_grid': {u'n_jobs': 4, u'max_tree_depth': {u'values': [8], u'gridMode': u'EXPLICIT', u'randomMode': u'RANGE', u'limit': {u'min': 1}, u'range': {u'scaling': u'LINEAR', u'max': 20, u'nbValues': 3, u'min': 10}}, u'selection_mode': u'auto', u'enabled': True, u'min_samples_leaf': {u'values': [1], u'gridMode': u'EXPLICIT', u'randomMode': u'RANGE', u'limit': {u'min': 1}, u'range': {u'scaling': u'LINEAR', u'max': 20, u'nbValues': 3, u'min': 1}}, u'n_estimators': {u'values': [100], u'gridMode': u'EXPLICIT', u'randomMode': u'RANGE', u'limit': {u'min': 1}, u'range': {u'scaling': u'LINEAR', u'max': 200, u'nbValues': 3, u'min': 80}}, u'max_feature_prop': {u'values': [0.3], u'gridMode': u'EXPLICIT', u'randomMode': u'RANGE', u'limit': {u'max': 1.0, u'min': 1e-23}, u'range': {u'scaling': u'LINEAR', u'max': 0.7, u'nbValues': 3, u'min': 0.1}}, u'max_features': {u'values': [5], u'gridMode': u'EXPLICIT', u'randomMode': u'RANGE', u'limit': {u'min': 1}, u'range': {u'scaling': u'LINEAR', u'max': 20, u'nbValues': 3, u'min': 1}}}, u'algorithm': u'RANDOM_FOREST_REGRESSION', u'grid_search_params': {u'nIter': 0, u'nJobs': 4, u'nContainers': 4, u'distributed': False, u'nIterRandom': 3, u'bayesianOptimizer': u'SCIKIT_OPTIMIZE', u'strategy': u'GRID', u'randomized': True, u'shuffleIterations': 1, u'seed': 0, u'mode': u'KFOLD', u'timeout': 0, u'splitRatio': 0.8, u'nFolds': 5, u'stratified': True}, u'autoOptimizeThreshold': True, u'gridLength': 1, u'metrics': {u'customEvaluationMetricNeedsProba': False, u'evaluationMetric': u'R2', u'liftPoint': 0.4, u'costMatrixWeights': {u'fnGain': 0.0, u'tpGain': 1.0, u'tnGain': 0.0, u'fpGain': -0.3}, u'customEvaluationMetricGIB': True}, u'forcedClassifierThreshold': 0.0, u'skipExpensiveReports': False, u'max_ensemble_nodes_serialized': 6000, u'pluginAlgoCustomGridSearch': False} for algorithm RANDOM_FOREST_REGRESSION
[2021-04-01 23:09:08,350] [4003/MainThread] [INFO] [dataiku.doctor.prediction.common] Using K-Fold CV with k=5
[2021-04-01 23:09:08,370] [4003/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluation_monitor] No distributed container configuration is available to run this search
[2021-04-01 23:09:08,371] [4003/MainThread] [INFO] [dataiku.doctor.crossval.search_runner] Got single-point space, not performing hyperparameter search
[2021-04-01 23:09:08,371] [4003/MainThread] [INFO] [dataiku.doctor.crossval.strategies.grid_search_strategy] Running GridSearchStrategy for hyperparameters space: <dataiku.doctor.prediction.common.TreesHyperparametersSpace object at 0x7f2116aca850>
[2021-04-01 23:09:08,372] [4003/MainThread] [INFO] [dataiku.doctor.prediction.common] Fitting model:
[2021-04-01 23:09:08,372] [4003/MainThread] [INFO] [dataiku.doctor.prediction.common]   Model is: RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=8,
           max_features=u'auto', max_leaf_nodes=None,
           min_impurity_decrease=0.0, min_impurity_split=None,
           min_samples_leaf=1, min_samples_split=3,
           min_weight_fraction_leaf=0.0, n_estimators=100, n_jobs=4,
           oob_score=False, random_state=1337, verbose=2, warm_start=False)
[2021-04-01 23:09:08,372] [4003/MainThread] [INFO] [dataiku.doctor.prediction.common]   train_X class: <type 'numpy.ndarray'>
[2021-04-01 23:09:08,372] [4003/MainThread] [INFO] [dataiku.doctor.prediction.common]   train_X shape: (79912, 19)
[2021-04-01 23:09:08,372] [4003/MainThread] [INFO] [dataiku.doctor.prediction.common]   train_y shape: (79912,)
[2021-04-01 23:09:08,372] [4003/MainThread] [INFO] [dataiku.doctor.prediction.common]   calibration enabled: a sub-sample of the train data has been saved for calibration
[Parallel(n_jobs=4)]: Using backend ThreadingBackend with 4 concurrent workers.
building tree 1 of 100
building tree 2 of 100
building tree 3 of 100building tree 4 of 100
building tree 5 of 100
building tree 6 of 100
building tree 7 of 100
building tree 8 of 100
building tree 9 of 100
building tree 10 of 100
building tree 11 of 100
building tree 12 of 100
building tree 13 of 100
building tree 14 of 100
building tree 15 of 100
building tree 16 of 100
building tree 17 of 100
building tree 18 of 100
building tree 19 of 100
building tree 20 of 100
building tree 21 of 100
building tree 22 of 100
building tree 23 of 100
building tree 24 of 100
building tree 25 of 100
building tree 26 of 100
building tree 27 of 100
building tree 28 of 100
building tree 29 of 100
building tree 30 of 100
building tree 31 of 100
building tree 32 of 100
building tree 33 of 100
building tree 34 of 100
building tree 35 of 100
building tree 36 of 100
[Parallel(n_jobs=4)]: Done  33 tasks      | elapsed:    1.0s
building tree 37 of 100
building tree 38 of 100
building tree 39 of 100
building tree 40 of 100
building tree 41 of 100
building tree 42 of 100
building tree 43 of 100
building tree 44 of 100
building tree 45 of 100
building tree 46 of 100
building tree 47 of 100
building tree 48 of 100
building tree 49 of 100
building tree 50 of 100
building tree 51 of 100
building tree 52 of 100
building tree 53 of 100
building tree 54 of 100
building tree 55 of 100
building tree 56 of 100
building tree 57 of 100
building tree 58 of 100
building tree 59 of 100
building tree 60 of 100
building tree 61 of 100
building tree 62 of 100
building tree 63 of 100
building tree 64 of 100
building tree 65 of 100
building tree 66 of 100
building tree 67 of 100
building tree 68 of 100
building tree 69 of 100
building tree 70 of 100
building tree 71 of 100
building tree 72 of 100
building tree 73 of 100
building tree 74 of 100
building tree 75 of 100
building tree 76 of 100
building tree 77 of 100
building tree 78 of 100
building tree 79 of 100
building tree 80 of 100
building tree 81 of 100
building tree 82 of 100
building tree 83 of 100
building tree 84 of 100
building tree 85 of 100
building tree 86 of 100
building tree 87 of 100
building tree 88 of 100
building tree 89 of 100
building tree 90 of 100
building tree 91 of 100
building tree 92 of 100
building tree 93 of 100
building tree 94 of 100
building tree 95 of 100
building tree 96 of 100
building tree 97 of 100
building tree 98 of 100
building tree 99 of 100
building tree 100 of 100
[Parallel(n_jobs=4)]: Done 100 out of 100 | elapsed:    3.3s finished
[2021-04-01 23:09:11,904] [4003/MainThread] [INFO] [dataiku.doctor.prediction.regression_fit] RF Params are {'warm_start': False, 'oob_score': False, 'n_jobs': 4, 'min_impurity_decrease': 0.0, 'verbose': 2, 'max_leaf_nodes': None, 'bootstrap': True, 'min_samples_leaf': 1, 'n_estimators': 100, 'min_samples_split': 3, 'min_weight_fraction_leaf': 0.0, 'criterion': 'mse', 'random_state': 1337, 'min_impurity_split': None, 'max_features': u'auto', 'max_depth': 8} 
[2021-04-01 23:09:11,904] [4003/MainThread] [INFO] [dataiku.doctor.prediction.regression_fit] Output params are {'resolved': {'rf': {'max_tree_depth': 8, 'selection_mode': u'auto', 'njobs': 4, 'min_samples_leaf': 1, 'estimators': 100}, 'skipExpensiveReports': False, 'algorithm': u'RANDOM_FOREST_REGRESSION'}, 'other': {'rf_min_samples_split': 3}}
[2021-04-01 23:09:12,163] [4003/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Fitting model
[2021-04-01 23:09:12,163] [4003/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Saving model
[2021-04-01 23:09:12,236] [4003/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Saving model
[2021-04-01 23:09:12,236] [4003/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Scoring model
[2021-04-01 23:09:12,237] [4003/MainThread] [INFO] [dataiku.doctor.prediction.regression_scoring] Intrinsic scoring
[2021-04-01 23:09:12,237] [4003/MainThread] [INFO] [dataiku.doctor.prediction.regression_scoring] Extracting rescalers
[2021-04-01 23:09:12,237] [4003/MainThread] [INFO] [dataiku.doctor.prediction.regression_scoring] Creating random forest trees summary
[2021-04-01 23:09:12,283] [4003/MainThread] [INFO] [dataiku.doctor.prediction.regression_scoring] Computing RF PDP
(100, 1)
[2021-04-01 23:09:12,442] [4003/MainThread] [INFO] [dataiku.doctor.prediction.scoring_base] Computing feature importance
[2021-04-01 23:09:12,703] [4003/MainThread] [INFO] [dataiku.doctor.prediction.common] prepare multiframe shape=(20088,19) tn=381672 nnz=180792 fill_ratio=0.47
[2021-04-01 23:09:12,703] [4003/MainThread] [INFO] [dataiku.doctor.prediction.common] too small, using array
[2021-04-01 23:09:12,704] [4003/MainThread] [INFO] [dataiku.doctor.prediction.regression_scoring] Creating predictions on test set
[Parallel(n_jobs=4)]: Using backend ThreadingBackend with 4 concurrent workers.
[Parallel(n_jobs=4)]: Done  33 tasks      | elapsed:    0.0s
[Parallel(n_jobs=4)]: Done 100 out of 100 | elapsed:    0.0s finished
[2021-04-01 23:09:12,842] [4003/MainThread] [INFO] [dataiku.doctor.prediction.regression_scoring] Computing regression performance on [0.2541558  0.2541558  0.2541558  ... 0.25508611 0.26947058 0.25365373]
[2021-04-01 23:09:13,286] [4003/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Scoring model
[2021-04-01 23:09:13,287] [4003/MainThread] [INFO] [root] Reading with dtypes: None
[2021-04-01 23:09:13,287] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_month: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-01 23:09:13,287] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_woy: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-01 23:09:13,287] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_dow: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-01 23:09:13,287] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for dep_hour: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-01 23:09:13,287] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for UniqueCarrier: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2021-04-01 23:09:13,287] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for FlightNum: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-01 23:09:13,287] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for AirTime: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-01 23:09:13,287] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Late: str (schema_type=boolean feature_type=CATEGORY feature_role=INPUT)
[2021-04-01 23:09:13,287] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Origin: None (schema_type=string feature_type=CATEGORY feature_role=REJECT)
[2021-04-01 23:09:13,287] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Dest: None (schema_type=string feature_type=CATEGORY feature_role=REJECT)
[2021-04-01 23:09:13,287] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Distance: <type 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2021-04-01 23:09:13,287] [4003/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Late_avg: <type 'numpy.float64'> (schema_type=double feature_type=NUMERIC feature_role=TARGET)
[2021-04-01 23:09:13,287] [4003/MainThread] [INFO] [root] Reading with FIXED dtypes: {u'dep_hour': <type 'numpy.float64'>, u'Late': 'str', u'UniqueCarrier': 'str', u'FlightNum': <type 'numpy.float64'>, u'Late_avg': <type 'numpy.float64'>, u'dep_woy': <type 'numpy.float64'>, u'Distance': <type 'numpy.float64'>, u'dep_month': <type 'numpy.float64'>, u'dep_dow': <type 'numpy.float64'>, u'AirTime': <type 'numpy.float64'>}
[2021-04-01 23:09:13,317] [4003/MainThread] [INFO] [root] Loaded table
[2021-04-01 23:09:13,318] [4003/MainThread] [INFO] [dataiku.doctor.prediction.background_rows_handler] Building background rows
[2021-04-01 23:09:13,324] [4003/MainThread] [INFO] [dataiku.doctor.prediction.background_rows_handler] Background estimated bias <= 0.00153913022213
[2021-04-01 23:09:13,328] [4003/MainThread] [INFO] [dataiku.doctor.prediction.background_rows_handler] Built background rows with shape=(1000, 12)
[2021-04-01 23:09:13,328] [4003/MainThread] [INFO] [dataiku.doctor.prediction.background_rows_handler] Saving background rows with shape=(1000, 12)
[2021-04-01 23:09:13,335] [4003/MainThread] [INFO] [dataiku.doctor.utils]  Coercion done
[2021-04-01 23:09:13,336] [4003/MainThread] [INFO] [dataiku.doctor.prediction.histogram_handler] Computing histograms
[2021-04-01 23:09:13,368] [4003/MainThread] [INFO] [dataiku.doctor.prediction.histogram_handler] Saving histograms
[2021-04-01 23:09:13,369] [4003/MainThread] [INFO] [dataiku.doctor.prediction.column_importance_handler] Computing column importance
[2021-04-01 23:09:13,369] [4003/MainThread] [INFO] [dataiku.doctor.prediction.column_importance_handler] Reusing the model feature importance
[2021-04-01 23:09:13,370] [4003/MainThread] [INFO] [dataiku.doctor.prediction.column_importance_handler] Saving column importance
[2021/04/01-23:09:13.375] [MRT-67774] [INFO] [dku.block.link.interaction]  - Check result for nullity exceptionIfNull=true result=not null
[2021-04-01 23:09:13,375] [4003/MainThread] [INFO] [dataiku.base.socket_block_link] Client closed
[2021/04/01-23:09:13.375] [MRT-67774] [INFO] [dku.analysis.prediction]  - Training returned ok
[2021/04/01-23:09:13.377] [MRT-67774] [INFO] [dku.ml.versioning]  - Dumping version info {"trainedWithDSSVersion":"9.0.0","trainedWithDSSConfVersion":"9000","trainedWithDSSBackend":"PY_MEMORY","backendCompatibilityVersion":18} in /data/home/dkuacademy/Library/dss-9.0/design/analysis-data/QSDE_SF_TO_FS_V2/oSylVxXB/0IfwLWGQ/sessions/s1/pp1/m1
[2021/04/01-23:09:13.379] [MRT-67774] [INFO] [dku.analysis.ml]  - preTrain={
  "algorithm": "RANDOM_FOREST_REGRESSION",
  "rf_regressor_grid": {
    "n_estimators": {
      "gridMode": "EXPLICIT",
      "randomMode": "RANGE",
      "limit": {
        "min": 1
      },
      "values": [
        100
      ],
      "range": {
        "min": 80,
        "max": 200,
        "nbValues": 3,
        "scaling": "LINEAR"
      }
    },
    "n_jobs": 4,
    "max_tree_depth": {
      "gridMode": "EXPLICIT",
      "randomMode": "RANGE",
      "limit": {
        "min": 1
      },
      "values": [
        8
      ],
      "range": {
        "min": 10,
        "max": 20,
        "nbValues": 3,
        "scaling": "LINEAR"
      }
    },
    "min_samples_leaf": {
      "gridMode": "EXPLICIT",
      "randomMode": "RANGE",
      "limit": {
        "min": 1
      },
      "values": [
        1
      ],
      "range": {
        "min": 1,
        "max": 20,
        "nbValues": 3,
        "scaling": "LINEAR"
      }
    },
    "selection_mode": "auto",
    "max_features": {
      "gridMode": "EXPLICIT",
      "randomMode": "RANGE",
      "limit": {
        "min": 1
      },
      "values": [
        5
      ],
      "range": {
        "min": 1,
        "max": 20,
        "nbValues": 3,
        "scaling": "LINEAR"
      }
    },
    "max_feature_prop": {
      "gridMode": "EXPLICIT",
      "randomMode": "RANGE",
      "limit": {
        "min": 1.0E-23,
        "max": 1.0
      },
      "values": [
        0.3
      ],
      "range": {
        "min": 0.1,
        "max": 0.7,
        "nbValues": 3,
        "scaling": "LINEAR"
      }
    },
    "enabled": true
  },
  "max_ensemble_nodes_serialized": 6000,
  "metrics": {
    "evaluationMetric": "R2",
    "customEvaluationMetricGIB": true,
    "customEvaluationMetricNeedsProba": false,
    "costMatrixWeights": {
      "tpGain": 1.0,
      "tnGain": 0.0,
      "fpGain": -0.3,
      "fnGain": 0.0
    },
    "liftPoint": 0.4
  },
  "autoOptimizeThreshold": true,
  "forcedClassifierThreshold": 0.0,
  "gridLength": 1,
  "grid_search_params": {
    "mode": "KFOLD",
    "splitRatio": 0.8,
    "shuffleIterations": 1,
    "nFolds": 5,
    "stratified": true,
    "strategy": "GRID",
    "bayesianOptimizer": "SCIKIT_OPTIMIZE",
    "randomized": true,
    "seed": 0,
    "nIter": 0,
    "nIterRandom": 3,
    "timeout": 0,
    "nJobs": 4,
    "distributed": false,
    "nContainers": 4
  },
  "pluginAlgoCustomGridSearch": false,
  "computeLearningCurves": false,
  "skipExpensiveReports": false
}
[2021/04/01-23:09:13.380] [MRT-67774] [INFO] [dku.analysis.ml]  - amp={
  "resolved": {
    "algorithm": "RANDOM_FOREST_REGRESSION",
    "rf": {
      "estimators": 100,
      "max_tree_depth": 8,
      "min_samples_leaf": 1,
      "selection_mode": "auto",
      "max_features": 0,
      "max_feature_prop": 0.0
    },
    "skipExpensiveReports": false
  },
  "other": {
    "rf_min_samples_split": 3
  }
}
[2021/04/01-23:09:13.380] [MRT-67774] [WARN] [dku.kernels]  - Killing kernel python-single-command-kernel
[2021/04/01-23:09:13.380] [MRT-67774] [INFO] [dku.security.process]  - Killing process PID: 4003
[2021/04/01-23:09:13.383] [MRT-67774] [INFO] [dip.tickets]  - Destroying API ticket for analysis-ml-QSDE_SF_TO_FS_V2-DEz4pzP on behalf of tara
[2021/04/01-23:09:13.395] [KNL-python-single-command-kernel-err-67782] [DEBUG] [process]  - StreamToLine: EOF (stream closed)
[2021/04/01-23:09:13.395] [KNL-python-single-command-kernel-out-67781] [DEBUG] [process]  - StreamToLine: EOF (stream closed)
[2021/04/01-23:09:13.396] [KNL-python-single-command-kernel-monitor-67779] [INFO] [dku.kernels]  - Process done with code 143
[2021/04/01-23:09:13.398] [KNL-python-single-command-kernel-monitor-67779] [WARN] [dku.resource]  - stat file for pid 4003 does not exist. Process died?
[2021/04/01-23:09:13.398] [KNL-python-single-command-kernel-monitor-67779] [INFO] [dku.resourceusage]  - Reporting completion of CRU:{"context":{"type":"ANALYSIS_ML_TRAIN","authIdentifier":"tara","projectKey":"QSDE_SF_TO_FS_V2","analysisId":"oSylVxXB","mlTaskId":"0IfwLWGQ","sessionId":"s1"},"type":"LOCAL_PROCESS","id":"vi4lq9dOAyaEglTq","startTime":1617311346653,"localProcess":{"pid":4003,"commandName":"/data/home/dkuacademy/Library/dss-9.0/design/bin/python","cpuUserTimeMS":14020,"cpuSystemTimeMS":770,"cpuChildrenUserTimeMS":0,"cpuChildrenSystemTimeMS":0,"cpuTotalMS":14790,"cpuCurrent":2.958,"vmSizeMB":2415,"vmRSSMB":186,"vmHWMMB":186,"vmRSSAnonMB":167,"vmDataMB":1753,"vmSizePeakMB":2479,"vmRSSPeakMB":186,"vmRSSTotalMBS":930,"majorFaults":90,"childrenMajorFaults":0}}
[2021/04/01-23:09:13.399] [MRT-67774] [INFO] [dku.block.link]  - Closed socket
[2021/04/01-23:09:13.399] [MRT-67774] [INFO] [dku.block.link]  - Closed socket
[2021/04/01-23:09:13.399] [MRT-67774] [INFO] [dku.block.link]  - Closed serverSocket
[2021/04/01-23:09:13.399] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.analysis.ml.python] T-0IfwLWGQ - Processing thread joined ...
[2021/04/01-23:09:13.399] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.analysis.ml.python] T-0IfwLWGQ - Joining processing thread ...
[2021/04/01-23:09:13.399] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.analysis.ml.python] T-0IfwLWGQ - Processing thread joined ...
[2021/04/01-23:09:13.399] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.analysis.prediction] T-0IfwLWGQ - Train done
[2021/04/01-23:09:13.399] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.analysis.prediction] T-0IfwLWGQ - Train done
[2021/04/01-23:09:13.402] [FT-TrainWorkThread-33cw3oHI-67773] [INFO] [dku.analysis.trainingdetails] T-0IfwLWGQ - Publishing mltask-train-done reflected event
