[2024/11/04-12:45:03.713] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.analysis.prediction]  - [ct: 0] ******************************************
[2024/11/04-12:45:03.713] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.analysis.prediction]  - [ct: 0] ** Start train session s1
[2024/11/04-12:45:03.713] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.analysis.prediction]  - [ct: 0] ******************************************
[2024/11/04-12:45:03.714] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.shaker.data] T-YgfwjSKZ - [ct: 1] Need to compute sampleId before checking memory cache
[2024/11/04-12:45:03.714] [FT-TrainWorkThread-04bwcSMM-109089] [DEBUG] [dip.shaker.runner] T-YgfwjSKZ - [ct: 1] Script settings sampleMax=-1 processedMax=-1
[2024/11/04-12:45:03.715] [FT-TrainWorkThread-04bwcSMM-109089] [DEBUG] [dip.shaker.runner] T-YgfwjSKZ - [ct: 2] Processing with sampleMax=-1 processedMax=-1
[2024/11/04-12:45:03.715] [FT-TrainWorkThread-04bwcSMM-109089] [DEBUG] [dip.shaker.runner] T-YgfwjSKZ - [ct: 2] Computed required sample id : 82253b4c2017db6f8aee04e4401b41ac-NA-fd778b1d8bbab9661a9d3800a481c6350--d751713988987e9331980363e24189ce
[2024/11/04-12:45:03.715] [FT-TrainWorkThread-04bwcSMM-109089] [DEBUG] [dku.shaker.cache] T-YgfwjSKZ - Shaker MemoryCache get on dataset DKU_TUT_CUSTOMMODELING.ecommerce_reviews key=ds=ba612b834cc4b4fda48355dedc48a6e7--scr=f5dc2b6423ca7914f5fec81cb888c7d3--samp=82253b4c2017db6f8aee04e4401b41ac-NA-fd778b1d8bbab9661a9d3800a481c6350--d751713988987e9331980363e24189ce: hit
[2024/11/04-12:45:03.715] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.shaker.schema] T-YgfwjSKZ - [ct: 2] Column Clothing ID meaning=LongMeaning fail=0
[2024/11/04-12:45:03.716] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.shaker.schema] T-YgfwjSKZ - [ct: 3] Column Age meaning=LongMeaning fail=0
[2024/11/04-12:45:03.716] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.shaker.schema] T-YgfwjSKZ - [ct: 3] Column Title meaning=FreeText fail=0
[2024/11/04-12:45:03.716] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.shaker.schema] T-YgfwjSKZ - [ct: 3] Column Review Text meaning=FreeText fail=0
[2024/11/04-12:45:03.716] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.shaker.schema] T-YgfwjSKZ - [ct: 3] Column Rating meaning=LongMeaning fail=0
[2024/11/04-12:45:03.716] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.shaker.schema] T-YgfwjSKZ - [ct: 3] Column Recommended IND meaning=LongMeaning fail=0
[2024/11/04-12:45:03.716] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.shaker.schema] T-YgfwjSKZ - [ct: 3] Column Positive Feedback Count meaning=LongMeaning fail=0
[2024/11/04-12:45:03.716] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.shaker.schema] T-YgfwjSKZ - [ct: 3] Column Division Name meaning=Text fail=0
[2024/11/04-12:45:03.717] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.shaker.schema] T-YgfwjSKZ - [ct: 4] Column Department Name meaning=Text fail=0
[2024/11/04-12:45:03.717] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.shaker.schema] T-YgfwjSKZ - [ct: 4] Column Class Name meaning=Text fail=0
[2024/11/04-12:45:03.720] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.datasets.file] T-YgfwjSKZ - [ct: 7] Building Filesystem handler config: {"path":"/data/home/samplemaster/dss_home_13/uploads/DKU_TUT_CUSTOMMODELING/datasets/ecommerce_reviews","notReadyIfEmpty":false,"filesSelectionRules":{"mode":"ALL","excludeRules":[],"includeRules":[],"explicitFiles":[]},"variablesExpansionLoopConfig":{"enabled":false,"mode":"CREATE_VARIABLE_FOR_EACH_COLUMN","replacements":[]}}
[2024/11/04-12:45:03.721] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.datasets.fsbased] T-YgfwjSKZ - [ct: 8] gss {"path":"/data/home/samplemaster/dss_home_13/uploads/DKU_TUT_CUSTOMMODELING/datasets/ecommerce_reviews","notReadyIfEmpty":false,"filesSelectionRules":{"mode":"ALL","excludeRules":[],"includeRules":[],"explicitFiles":[]},"variablesExpansionLoopConfig":{"enabled":false,"mode":"CREATE_VARIABLE_FOR_EACH_COLUMN","replacements":[]}}
[2024/11/04-12:45:03.721] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.datasets.ftplike] T-YgfwjSKZ - Variable-expanded selection rules: {"mode":"ALL","excludeRules":[],"includeRules":[],"explicitFiles":[]}
[2024/11/04-12:45:03.721] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.datasets.ftplike] T-YgfwjSKZ - Enumerating Filesystem dataset prefix=
[2024/11/04-12:45:03.721] [FT-TrainWorkThread-04bwcSMM-109089] [DEBUG] [dku.datasets.fsbased] T-YgfwjSKZ - [ct: 8] Building FS provider for dataset handler: DKU_TUT_CUSTOMMODELING.ecommerce_reviews
[2024/11/04-12:45:03.721] [FT-TrainWorkThread-04bwcSMM-109089] [DEBUG] [dku.datasets.fsbased] T-YgfwjSKZ - [ct: 8] FS Provider built
[2024/11/04-12:45:03.722] [FT-TrainWorkThread-04bwcSMM-109089] [DEBUG] [dku.fs.local] T-YgfwjSKZ - [ct: 9] Enumerating local filesystem prefix=/
[2024/11/04-12:45:03.722] [FT-TrainWorkThread-04bwcSMM-109089] [DEBUG] [dku.fs.local] T-YgfwjSKZ - [ct: 9] Enumeration done nb_paths=1 size=8316774
[2024/11/04-12:45:03.722] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.input.push] T-YgfwjSKZ - USTP: push selection.method=HEAD_SEQUENTIAL records=100000 ratio=0.02 col=null
[2024/11/04-12:45:03.722] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.format] T-YgfwjSKZ - [ct: 9] Extractor run: limit={"maxBytes":-1,"maxRecords":100000,"ordering":{"enabled":false,"rules":[]}} totalRecords=0
[2024/11/04-12:45:03.722] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.format] T-YgfwjSKZ - [ct: 9] Start uncompressed stream: /data/home/samplemaster/dss_home_13/uploads/DKU_TUT_CUSTOMMODELING/datasets/ecommerce_reviews/ecommerce_reviews.csv / totalRecsBefore=0
[2024/11/04-12:45:04.112] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.format] T-YgfwjSKZ - [ct: 399] after stream totalComp=8316774 totalUncomp=8316774 totalRec=22641
[2024/11/04-12:45:04.112] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.format] T-YgfwjSKZ - [ct: 399] Extractor run done, totalCompressed=8316774 totalRecords=22641
[2024/11/04-12:45:04.113] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.analysis.splits] T-YgfwjSKZ - [ct: 400] Checking if splits are up to date. Policy: type=SPLIT_SINGLE_DATASET,split=RANDOM,splitBeforePrepare=true,ds=ecommerce_reviews,sel=(method=head-s,records=100000),r=0.8,s=1337, instance id: f7d03991ff273ff585ab94baa468252a-0
[2024/11/04-12:45:04.115] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.analysis.ml.python] T-YgfwjSKZ - [ct: 402] Joining processing thread ...
[2024/11/04-12:45:04.115] [MRT-109090] [INFO] [dku.analysis.ml.python]  - Running a preprocessing set: pp1 in /data/home/samplemaster/dss_home_13/analysis-data/DKU_TUT_CUSTOMMODELING/lTMJmloT/YgfwjSKZ/sessions/s1/pp1
[2024/11/04-12:45:04.116] [MRT-109090] [INFO] [dku.block.link]  - Started a socket on port 43685
[2024/11/04-12:45:04.116] [MRT-109090] [INFO] [com.dataiku.dip.analysis.ml.distributed.workers]  - New worker pool created: pool-q2by1oxtzsyjvkhg
[2024/11/04-12:45:04.117] [MRT-109090] [INFO] [dku.python.single_command.kernel]  - Attached worker pool to com.dataiku.dip.analysis.coreservices.AnalysisMLKernel kernel: pool-q2by1oxtzsyjvkhg
[2024/11/04-12:45:04.117] [MRT-109090] [INFO] [dku.ml.kernel]  - Writing output of python-single-command-kernel to /data/home/samplemaster/dss_home_13/analysis-data/DKU_TUT_CUSTOMMODELING/lTMJmloT/YgfwjSKZ/sessions/s1/pp1/train.log
[2024/11/04-12:45:04.117] [MRT-109090] [INFO] [dku.code.envs.resolution]  - Executing Python activity in builtin env
[2024/11/04-12:45:04.118] [MRT-109090] [INFO] [dku.python.single_command.kernel]  - Starting Python process for kernel  python-single-command-kernel
[2024/11/04-12:45:04.118] [MRT-109090] [INFO] [dip.tickets]  - Creating API ticket for analysis-ml-DKU_TUT_CUSTOMMODELING-fY6u00U on behalf of admin id=analysis-ml-DKU_TUT_CUSTOMMODELING-fY6u00U_A6xP5j5mgSVi
[2024/11/04-12:45:04.118] [MRT-109090] [INFO] [dku.process]  - Starting process (regular)
[2024/11/04-12:45:04.120] [MRT-109090] [INFO] [dku.process]  - Process started with pid=14336
[2024/11/04-12:45:04.120] [KNL-python-single-command-kernel-monitor-109098] [DEBUG] [dku.resourceusage]  - Reporting start of CRU:{"context":{"type":"ANALYSIS_ML_TRAIN","authIdentifier":"admin","projectKey":"DKU_TUT_CUSTOMMODELING","analysisId":"lTMJmloT","mlTaskId":"YgfwjSKZ","sessionId":"s1"},"type":"LOCAL_PROCESS","id":"78OomSa0uoDB0F0N","startTime":1730724304120,"localProcess":{"cpuCurrent":0.0,"cpuAverageOverPast60Seconds":0.0}}
[2024/11/04-12:45:04.121] [process-resource-monitor-14336-109102] [DEBUG] [dku.resource]  - Process stats for pid 14336: {"pid":14336,"commandName":"/data/home/samplemaster/dss_home_13/bin/python","cpuUserTimeMS":0,"cpuSystemTimeMS":0,"cpuChildrenUserTimeMS":0,"cpuChildrenSystemTimeMS":0,"cpuTotalMS":0,"cpuCurrent":0.0,"cpuAverageOverPast60Seconds":0.0,"vmSizeMB":24,"vmRSSMB":0,"vmHWMMB":0,"vmRSSAnonMB":0,"vmDataMB":0,"vmSizePeakMB":24,"vmRSSPeakMB":0,"vmRSSTotalMBS":0,"majorFaults":2,"childrenMajorFaults":0}
Installing debugging signal handler
[2024-11-04 12:45:04,840] [14336/MainThread] [INFO] [dataiku.base.socket_block_link] no cert found
[2024-11-04 12:45:04,840] [14336/MainThread] [INFO] [dataiku.base.socket_block_link] Connecting to localhost (127.0.0.1) at port 43685
[2024-11-04 12:45:04,840] [14336/MainThread] [INFO] [dataiku.base.socket_block_link] Connected to localhost (127.0.0.1) at port 43685
[2024/11/04-12:45:04.841] [MRT-109090] [INFO] [dku.link.secret_protected]  - Connected to kernel
[2024/11/04-12:45:04.843] [MRT-109090] [INFO] [dku.block.link.interaction]  - Execute link command respClazz=true respTypeToken=false respIsString=false is=false asyncInputStream=false os=false
[2024-11-04 12:45:04,936] [14336/MainThread] [INFO] [root] Running analysis command: train_prediction_models_nosave
[2024-11-04 12:45:04,936] [14336/MainThread] [INFO] [dataiku.doctor.utils.gpu_execution] Not logging `nvidia-smi`: `useGPU` is False
[2024-11-04 12:45:04,936] [14336/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: DatasetSanityCheckDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_DATASET_SANITY_CHECKS
[2024-11-04 12:45:04,936] [14336/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: ClassifierAccuracyCheckDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_MODEL_CHECK
[2024-11-04 12:45:04,936] [14336/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: RegressionR2CheckDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_MODEL_CHECK
[2024-11-04 12:45:04,936] [14336/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: LeakageDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_LEAKAGE_DETECTION
[2024-11-04 12:45:04,936] [14336/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: TreeOverfitDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_TRAINING_OVERFIT
[2024-11-04 12:45:04,936] [14336/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: MLAssertionsDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_ML_ASSERTIONS
[2024-11-04 12:45:04,936] [14336/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: ModelCheckDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_MODEL_CHECK
[2024-11-04 12:45:04,936] [14336/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] enabling diagnostic callback: AbnormalPredictionsDetectionDiagnostic of type DiagnosticType.ML_DIAGNOSTICS_ABNORMAL_PREDICTIONS_DETECTION
[2024-11-04 12:45:04,936] [14336/MainThread] [INFO] [dataiku.doctor.commands] PPS is {'feature_selection_params': {'method': 'NONE', 'random_forest_params': {'n_trees': 30, 'depth': 10, 'n_features': 25}, 'lasso_params': {'alpha': [0.01, 0.1, 1.0, 10.0, 100.0], 'cross_validate': True}, 'pca_params': {'n_features': 25, 'variance_proportion': 0.9}, 'correlation_params': {'min_abs_correlation': 0.0, 'max_abs_correlation': 1.0, 'n_features': 25}, 'custom_params': {'code': '# type your code here'}}, 'preprocessingFitSampleRatio': 1.0, 'preprocessingFitSampleSeed': 1337, 'target_remapping': [{'sourceValue': '5', 'mappedValue': 0, 'sampleFreq': 5527}, {'sourceValue': '4', 'mappedValue': 1, 'sampleFreq': 2184}, {'sourceValue': '3', 'mappedValue': 2, 'sampleFreq': 1248}, {'sourceValue': '2', 'mappedValue': 3, 'sampleFreq': 682}, {'sourceValue': '1', 'mappedValue': 4, 'sampleFreq': 359}], 'skipPreprocessing': False, 'per_feature': {'Recommended IND': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'keep_regular': False, 'rescaling': 'AVGSTD', 'monotonic': 'NONE', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'datetime_cyclical_periods': [], 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Division Name': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 100, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'NONE', 'impact_method': 'M_ESTIMATOR', 'impact_m': 10, 'impact_kfold': True, 'impact_kfold_k': 5, 'impact_kfold_seed': 1337, 'ordinal_order': 'COUNT', 'ordinal_ascending': False, 'ordinal_default_mode': 'HIGHEST', 'ordinal_default_value': 0, 'frequency_default_mode': 'EXPLICIT', 'frequency_default_value': 0.0, 'frequency_normalized': True, 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Text'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Department Name': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 100, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'NONE', 'impact_method': 'M_ESTIMATOR', 'impact_m': 10, 'impact_kfold': True, 'impact_kfold_k': 5, 'impact_kfold_seed': 1337, 'ordinal_order': 'COUNT', 'ordinal_ascending': False, 'ordinal_default_mode': 'HIGHEST', 'ordinal_default_value': 0, 'frequency_default_mode': 'EXPLICIT', 'frequency_default_value': 0.0, 'frequency_normalized': True, 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Text'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Positive Feedback Count': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'keep_regular': False, 'rescaling': 'AVGSTD', 'monotonic': 'NONE', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'datetime_cyclical_periods': [], 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Review Text': {'text_handling': 'TOKENIZE_HASHING_SVD', 'minRowsRatio': 0.001, 'maxRowsRatio': 0.8, 'maxWords': 0, 'ngramMinSize': 1, 'ngramMaxSize': 1, 'hashSize': 200000, 'hashSVDSVDLimit': 50000, 'hashSVDSVDComponents': 100, 'stopWordsMode': 'NONE', 'maxSequenceLength': 128, 'sentenceEmbeddingBatchSize': 32, 'useCustomVectorizer': False, 'isStructuredRef': False, 'name': 'Review Text', 'role': 'REJECT', 'type': 'TEXT', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'FreeText'}, 'autoReason': 'REJECT_DEFAULT_TEXT_HANDLING', 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Rating': {'generate_derivative': False, 'impute_constant_value': 0.0, 'keep_regular': False, 'monotonic': 'NONE', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'datetime_cyclical_periods': [], 'role': 'TARGET', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Class Name': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 100, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'NONE', 'impact_method': 'M_ESTIMATOR', 'impact_m': 10, 'impact_kfold': True, 'impact_kfold_k': 5, 'impact_kfold_seed': 1337, 'ordinal_order': 'COUNT', 'ordinal_ascending': False, 'ordinal_default_mode': 'HIGHEST', 'ordinal_default_value': 0, 'frequency_default_mode': 'EXPLICIT', 'frequency_default_value': 0.0, 'frequency_normalized': True, 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Text'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Title': {'text_handling': 'TOKENIZE_HASHING_SVD', 'minRowsRatio': 0.001, 'maxRowsRatio': 0.8, 'maxWords': 0, 'ngramMinSize': 1, 'ngramMaxSize': 1, 'hashSize': 200000, 'hashSVDSVDLimit': 50000, 'hashSVDSVDComponents': 100, 'stopWordsMode': 'NONE', 'maxSequenceLength': 128, 'sentenceEmbeddingBatchSize': 32, 'useCustomVectorizer': False, 'isStructuredRef': False, 'name': 'Title', 'role': 'REJECT', 'type': 'TEXT', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'FreeText'}, 'autoReason': 'REJECT_DEFAULT_TEXT_HANDLING', 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Clothing ID': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'keep_regular': False, 'rescaling': 'AVGSTD', 'monotonic': 'NONE', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'datetime_cyclical_periods': [], 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Age': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'keep_regular': False, 'rescaling': 'AVGSTD', 'monotonic': 'NONE', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'datetime_cyclical_periods': [], 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}}, 'reduce': {'enabled': False, 'kept_variance': 0.0}, 'feature_generation': {'pairwise_linear': {'behavior': 'DISABLED'}, 'polynomial_combinations': {'behavior': 'DISABLED'}, 'manual_interactions': {'interactions': []}, 'numericals_clustering': {'k': 0, 'all_features': False, 'input_features': [], 'behavior': 'DISABLED'}, 'categoricals_count_transformer': {'all_features': False, 'input_features': [], 'behavior': 'DISABLED'}}}
[2024-11-04 12:45:04,936] [14336/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Loading train set
[2024-11-04 12:45:04,937] [14336/MainThread] [INFO] [root] Reading with dtypes: None
[2024-11-04 12:45:04,937] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Clothing ID: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2024-11-04 12:45:04,937] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Age: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2024-11-04 12:45:04,937] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Title: None (schema_type=string feature_type=TEXT feature_role=REJECT)
[2024-11-04 12:45:04,937] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Review Text: None (schema_type=string feature_type=TEXT feature_role=REJECT)
[2024-11-04 12:45:04,937] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Rating: <class 'numpy.object_'> (schema_type=bigint feature_type=NUMERIC feature_role=TARGET)
[2024-11-04 12:45:04,937] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Recommended IND: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2024-11-04 12:45:04,937] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Positive Feedback Count: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2024-11-04 12:45:04,937] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Division Name: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2024-11-04 12:45:04,937] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Department Name: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2024-11-04 12:45:04,937] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Class Name: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2024-11-04 12:45:04,937] [14336/MainThread] [INFO] [root] Reading with FIXED dtypes: {'Clothing ID': <class 'numpy.float64'>, 'Age': <class 'numpy.float64'>, 'Rating': <class 'numpy.object_'>, 'Recommended IND': <class 'numpy.float64'>, 'Positive Feedback Count': <class 'numpy.float64'>, 'Division Name': 'str', 'Department Name': 'str', 'Class Name': 'str'}
[2024-11-04 12:45:04,997] [14336/MainThread] [INFO] [root] Loaded table
[2024-11-04 12:45:04,998] [14336/MainThread] [INFO] [dataiku.doctor.utils] Coercion done
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] Adding diagnostic: <DoctorDiagnostic ML_DIAGNOSTICS_DATASET_SANITY_CHECKS: The train dataset is imbalanced (class '1' is only represented in 3.69% of rows; a well balanced dataset would contain ~20.00%), metrics can be misleading>
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [dataiku.doctor.utils.split] Loaded train df: shape=(18138,10)
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [dataiku.doctor.commands] Train col : Clothing ID (float64)
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [dataiku.doctor.commands] Train col : Age (float64)
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [dataiku.doctor.commands] Train col : Title (object)
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [dataiku.doctor.commands] Train col : Review Text (object)
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [dataiku.doctor.commands] Train col : Rating (object)
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [dataiku.doctor.commands] Train col : Recommended IND (float64)
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [dataiku.doctor.commands] Train col : Positive Feedback Count (float64)
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [dataiku.doctor.commands] Train col : Division Name (object)
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [dataiku.doctor.commands] Train col : Department Name (object)
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [dataiku.doctor.commands] Train col : Class Name (object)
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Loading train set
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Loading test set
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [root] Reading with dtypes: None
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Clothing ID: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Age: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Title: None (schema_type=string feature_type=TEXT feature_role=REJECT)
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Review Text: None (schema_type=string feature_type=TEXT feature_role=REJECT)
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Rating: <class 'numpy.object_'> (schema_type=bigint feature_type=NUMERIC feature_role=TARGET)
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Recommended IND: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Positive Feedback Count: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Division Name: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Department Name: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Class Name: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2024-11-04 12:45:04,999] [14336/MainThread] [INFO] [root] Reading with FIXED dtypes: {'Clothing ID': <class 'numpy.float64'>, 'Age': <class 'numpy.float64'>, 'Rating': <class 'numpy.object_'>, 'Recommended IND': <class 'numpy.float64'>, 'Positive Feedback Count': <class 'numpy.float64'>, 'Division Name': 'str', 'Department Name': 'str', 'Class Name': 'str'}
[2024-11-04 12:45:05,015] [14336/MainThread] [INFO] [root] Loaded table
[2024-11-04 12:45:05,015] [14336/MainThread] [INFO] [dataiku.doctor.utils] Coercion done
[2024-11-04 12:45:05,016] [14336/MainThread] [INFO] [dataiku.doctor.diagnostics.diagnostics] Adding diagnostic: <DoctorDiagnostic ML_DIAGNOSTICS_DATASET_SANITY_CHECKS: The test dataset is imbalanced (class '1' is only represented in 3.38% of rows; a well balanced dataset would contain ~20.00%), metrics can be misleading>
[2024-11-04 12:45:05,016] [14336/MainThread] [INFO] [dataiku.doctor.utils.split] Loaded test df: shape=(4503,10)
[2024-11-04 12:45:05,016] [14336/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Loading test set
[2024-11-04 12:45:05,016] [14336/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Collecting statistics
[2024-11-04 12:45:05,016] [14336/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Recommended IND... (type=NUMERIC)
[2024-11-04 12:45:05,016] [14336/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2024-11-04 12:45:05,018] [14336/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Division Name... (type=CATEGORY)
[2024-11-04 12:45:05,020] [14336/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Department Name... (type=CATEGORY)
[2024-11-04 12:45:05,022] [14336/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Positive Feedback Count... (type=NUMERIC)
[2024-11-04 12:45:05,022] [14336/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2024-11-04 12:45:05,022] [14336/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Review Text... (type=TEXT)
[2024-11-04 12:45:05,022] [14336/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Rating... (type=NUMERIC)
[2024-11-04 12:45:05,022] [14336/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Class Name... (type=CATEGORY)
[2024-11-04 12:45:05,024] [14336/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Title... (type=TEXT)
[2024-11-04 12:45:05,024] [14336/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Clothing ID... (type=NUMERIC)
[2024-11-04 12:45:05,024] [14336/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2024-11-04 12:45:05,025] [14336/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Looking at Age... (type=NUMERIC)
[2024-11-04 12:45:05,025] [14336/MainThread] [INFO] [dataiku.doctor.preprocessing_collector] Checking series of type: float64 (isM8=False)
[2024-11-04 12:45:05,026] [14336/MainThread] [INFO] [dataiku.doctor.preprocessing_handler] generating interactions
[2024-11-04 12:45:05,026] [14336/MainThread] [INFO] [dataiku.doctor.preprocessing_handler] {'feature_selection_params': {'method': 'NONE', 'random_forest_params': {'n_trees': 30, 'depth': 10, 'n_features': 25}, 'lasso_params': {'alpha': [0.01, 0.1, 1.0, 10.0, 100.0], 'cross_validate': True}, 'pca_params': {'n_features': 25, 'variance_proportion': 0.9}, 'correlation_params': {'min_abs_correlation': 0.0, 'max_abs_correlation': 1.0, 'n_features': 25}, 'custom_params': {'code': '# type your code here'}}, 'preprocessingFitSampleRatio': 1.0, 'preprocessingFitSampleSeed': 1337, 'target_remapping': [{'sourceValue': '5', 'mappedValue': 0, 'sampleFreq': 5527}, {'sourceValue': '4', 'mappedValue': 1, 'sampleFreq': 2184}, {'sourceValue': '3', 'mappedValue': 2, 'sampleFreq': 1248}, {'sourceValue': '2', 'mappedValue': 3, 'sampleFreq': 682}, {'sourceValue': '1', 'mappedValue': 4, 'sampleFreq': 359}], 'skipPreprocessing': False, 'per_feature': {'Recommended IND': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'keep_regular': False, 'rescaling': 'AVGSTD', 'monotonic': 'NONE', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'datetime_cyclical_periods': [], 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Division Name': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 100, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'NONE', 'impact_method': 'M_ESTIMATOR', 'impact_m': 10, 'impact_kfold': True, 'impact_kfold_k': 5, 'impact_kfold_seed': 1337, 'ordinal_order': 'COUNT', 'ordinal_ascending': False, 'ordinal_default_mode': 'HIGHEST', 'ordinal_default_value': 0, 'frequency_default_mode': 'EXPLICIT', 'frequency_default_value': 0.0, 'frequency_normalized': True, 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Text'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Department Name': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 100, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'NONE', 'impact_method': 'M_ESTIMATOR', 'impact_m': 10, 'impact_kfold': True, 'impact_kfold_k': 5, 'impact_kfold_seed': 1337, 'ordinal_order': 'COUNT', 'ordinal_ascending': False, 'ordinal_default_mode': 'HIGHEST', 'ordinal_default_value': 0, 'frequency_default_mode': 'EXPLICIT', 'frequency_default_value': 0.0, 'frequency_normalized': True, 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Text'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Positive Feedback Count': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'keep_regular': False, 'rescaling': 'AVGSTD', 'monotonic': 'NONE', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'datetime_cyclical_periods': [], 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Review Text': {'text_handling': 'TOKENIZE_HASHING_SVD', 'minRowsRatio': 0.001, 'maxRowsRatio': 0.8, 'maxWords': 0, 'ngramMinSize': 1, 'ngramMaxSize': 1, 'hashSize': 200000, 'hashSVDSVDLimit': 50000, 'hashSVDSVDComponents': 100, 'stopWordsMode': 'NONE', 'maxSequenceLength': 128, 'sentenceEmbeddingBatchSize': 32, 'useCustomVectorizer': False, 'isStructuredRef': False, 'name': 'Review Text', 'role': 'REJECT', 'type': 'TEXT', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'FreeText'}, 'autoReason': 'REJECT_DEFAULT_TEXT_HANDLING', 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Rating': {'generate_derivative': False, 'impute_constant_value': 0.0, 'keep_regular': False, 'monotonic': 'NONE', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'datetime_cyclical_periods': [], 'role': 'TARGET', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Class Name': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 100, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'NONE', 'impact_method': 'M_ESTIMATOR', 'impact_m': 10, 'impact_kfold': True, 'impact_kfold_k': 5, 'impact_kfold_seed': 1337, 'ordinal_order': 'COUNT', 'ordinal_ascending': False, 'ordinal_default_mode': 'HIGHEST', 'ordinal_default_value': 0, 'frequency_default_mode': 'EXPLICIT', 'frequency_default_value': 0.0, 'frequency_normalized': True, 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Text'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Title': {'text_handling': 'TOKENIZE_HASHING_SVD', 'minRowsRatio': 0.001, 'maxRowsRatio': 0.8, 'maxWords': 0, 'ngramMinSize': 1, 'ngramMaxSize': 1, 'hashSize': 200000, 'hashSVDSVDLimit': 50000, 'hashSVDSVDComponents': 100, 'stopWordsMode': 'NONE', 'maxSequenceLength': 128, 'sentenceEmbeddingBatchSize': 32, 'useCustomVectorizer': False, 'isStructuredRef': False, 'name': 'Title', 'role': 'REJECT', 'type': 'TEXT', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'FreeText'}, 'autoReason': 'REJECT_DEFAULT_TEXT_HANDLING', 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Clothing ID': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'keep_regular': False, 'rescaling': 'AVGSTD', 'monotonic': 'NONE', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'datetime_cyclical_periods': [], 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Age': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'keep_regular': False, 'rescaling': 'AVGSTD', 'monotonic': 'NONE', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'datetime_cyclical_periods': [], 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}}, 'reduce': {'enabled': False, 'kept_variance': 0.0}, 'feature_generation': {'pairwise_linear': {'behavior': 'DISABLED'}, 'polynomial_combinations': {'behavior': 'DISABLED'}, 'manual_interactions': {'interactions': []}, 'numericals_clustering': {'k': 0, 'all_features': False, 'input_features': [], 'behavior': 'DISABLED'}, 'categoricals_count_transformer': {'all_features': False, 'input_features': [], 'behavior': 'DISABLED'}}}
[2024-11-04 12:45:05,026] [14336/MainThread] [INFO] [dataiku.doctor.preprocessing_handler] No feature selection to perform
[2024-11-04 12:45:05,026] [14336/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Collecting statistics
[2024-11-04 12:45:05,026] [14336/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Preprocessing train set
[2024-11-04 12:45:05,026] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 18138
[2024-11-04 12:45:05,026] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RemapValueToOutput
[2024-11-04 12:45:05,028] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:MultipleImputeMissingFromInput
[2024-11-04 12:45:05,028] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {'Recommended IND': 0.8202668430918514, 'Positive Feedback Count': 2.6301135737126473, 'Clothing ID': 919.3338295291653, 'Age': 43.30047414268387}
[2024-11-04 12:45:05,029] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:MultipleImputeMissingFromInput
[2024-11-04 12:45:05,029] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2024-11-04 12:45:05,029] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (Recommended IND)
[2024-11-04 12:45:05,029] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale Recommended IND (avg=0.8202668430918514 std=0.38397562143486647 shift=0.8202668430918514 inv_scale=2.60433200488909)
[2024-11-04 12:45:05,033] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled Recommended IND (avg=8.8105360108360915e-16 std=0.9999999999999011) nulls=0
[2024-11-04 12:45:05,033] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (Positive Feedback Count)
[2024-11-04 12:45:05,033] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale Positive Feedback Count (avg=2.6301135737126473 std=5.792964725725614 shift=2.6301135737126473 inv_scale=0.17262318128041806)
[2024-11-04 12:45:05,034] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled Positive Feedback Count (avg=-3.7634222321123124e-16 std=1.0000000000000597) nulls=0
[2024-11-04 12:45:05,034] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (Clothing ID)
[2024-11-04 12:45:05,034] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale Clothing ID (avg=919.3338295291653 std=201.62590249681836 shift=919.3338295291653 inv_scale=0.004959680217752676)
[2024-11-04 12:45:05,035] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled Clothing ID (avg=-3.7809282297439586e-17 std=0.999999999999992) nulls=0
[2024-11-04 12:45:05,035] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RescalingProcessor2 (Age)
[2024-11-04 12:45:05,035] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale Age (avg=43.30047414268387 std=12.327406722157297 shift=43.30047414268387 inv_scale=0.08112006219463812)
[2024-11-04 12:45:05,035] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled Age (avg=1.5535042653537586e-17 std=1.0000000000000417) nulls=0
[2024-11-04 12:45:05,035] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FlushDFBuilder(num_flagonly)
[2024-11-04 12:45:05,036] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FlushDFBuilder(datetime_cyclical)
[2024-11-04 12:45:05,036] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FastSparseDummifyProcessor (Division Name)
[2024-11-04 12:45:05,039] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(18138, 5) nnz=18138
[2024-11-04 12:45:05,039] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FastSparseDummifyProcessor (Department Name)
[2024-11-04 12:45:05,042] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(18138, 8) nnz=18138
[2024-11-04 12:45:05,042] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FastSparseDummifyProcessor (Class Name)
[2024-11-04 12:45:05,045] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(18138, 22) nnz=18138
[2024-11-04 12:45:05,045] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:MultipleImputeMissingFromInput
[2024-11-04 12:45:05,045] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2024-11-04 12:45:05,045] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FlushDFBuilder(cat_flagpresence)
[2024-11-04 12:45:05,045] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:MultipleImputeMissingFromInput
[2024-11-04 12:45:05,045] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2024-11-04 12:45:05,045] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:MultipleImputeMissingFromInput
[2024-11-04 12:45:05,045] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2024-11-04 12:45:05,045] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:FlushDFBuilder(interaction)
[2024-11-04 12:45:05,045] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:RealignTarget
[2024-11-04 12:45:05,045] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] Realign target series = (18138,)
[2024-11-04 12:45:05,046] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] After realign target: (18138,)
[2024-11-04 12:45:05,046] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:DropRowsWhereNoTarget
[2024-11-04 12:45:05,046] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] Deleting 0 rows because one of ['target'] is missing or infinity
[2024-11-04 12:45:05,046] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] MF before = (18138, 39)
[2024-11-04 12:45:05,046] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] target before = (18138,)
[2024-11-04 12:45:05,046] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] MultiFrame, dropping rows: []
[2024-11-04 12:45:05,049] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] After DRWNT input_df=(18138, 10)
[2024-11-04 12:45:05,049] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] MF after = (18138, 39)
[2024-11-04 12:45:05,049] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] target after = (18138,)
[2024-11-04 12:45:05,049] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:DumpPipelineState
[2024-11-04 12:45:05,050] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] ********* Pipeline state (Before feature selection)
[2024-11-04 12:45:05,050] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]    input_df= (18138, 10) 
[2024-11-04 12:45:05,050] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]    current_mf=(18138, 39) 
[2024-11-04 12:45:05,050] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]    PPR: 
[2024-11-04 12:45:05,050] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]       target = <class 'pandas.core.series.Series'> ((18138,))
[2024-11-04 12:45:05,050] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:EmitCurrentMFAsResult
[2024-11-04 12:45:05,050] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 18138
[2024-11-04 12:45:05,050] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] FIT/PROCESS WITH Step:DumpPipelineState
[2024-11-04 12:45:05,050] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] ********* Pipeline state (At end)
[2024-11-04 12:45:05,050] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]    input_df= (18138, 10) 
[2024-11-04 12:45:05,050] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]    current_mf=(0, 0) 
[2024-11-04 12:45:05,050] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]    PPR: 
[2024-11-04 12:45:05,050] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]       target = <class 'pandas.core.series.Series'> ((18138,))
[2024-11-04 12:45:05,050] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]       TRAIN = <class 'dataiku.doctor.multiframe.MultiFrame'> ((18138, 39))
[2024-11-04 12:45:05,050] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]       UNPROCESSED = <class 'pandas.core.frame.DataFrame'> ((18138, 10))
[2024-11-04 12:45:05,050] [14336/MainThread] [INFO] [root] train multiframe initial size: 18138
[2024-11-04 12:45:05,050] [14336/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Preprocessing train set
[2024-11-04 12:45:05,050] [14336/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Preprocessing test set
[2024-11-04 12:45:05,050] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 4503
[2024-11-04 12:45:05,050] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RemapValueToOutput
[2024-11-04 12:45:05,051] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:MultipleImputeMissingFromInput
[2024-11-04 12:45:05,051] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {'Recommended IND': 0.8202668430918514, 'Positive Feedback Count': 2.6301135737126473, 'Clothing ID': 919.3338295291653, 'Age': 43.30047414268387}
[2024-11-04 12:45:05,052] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:MultipleImputeMissingFromInput
[2024-11-04 12:45:05,052] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2024-11-04 12:45:05,052] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (Recommended IND)
[2024-11-04 12:45:05,052] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale Recommended IND (avg=0.8132356206973129 std=0.38976554851485246 shift=0.8202668430918514 inv_scale=2.60433200488909)
[2024-11-04 12:45:05,052] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled Recommended IND (avg=-0.01831163751558812 std=1.0150788924003749) nulls=0
[2024-11-04 12:45:05,052] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (Positive Feedback Count)
[2024-11-04 12:45:05,052] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale Positive Feedback Count (avg=2.632467244059516 std=5.759328540308391 shift=2.6301135737126473 inv_scale=0.17262318128041806)
[2024-11-04 12:45:05,052] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled Positive Feedback Count (avg=0.0004062980629617316 std=0.9941936146671286) nulls=0
[2024-11-04 12:45:05,052] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (Clothing ID)
[2024-11-04 12:45:05,053] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale Clothing ID (avg=919.3264490339774 std=204.85099422656134 shift=919.3338295291653 inv_scale=0.004959680217752676)
[2024-11-04 12:45:05,053] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled Clothing ID (avg=-3.6604895980930003e-05 std=1.0159954236524484) nulls=0
[2024-11-04 12:45:05,053] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (Age)
[2024-11-04 12:45:05,053] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale Age (avg=43.19942260715079 std=12.32629568008881 shift=43.30047414268387 inv_scale=0.08112006219463812)
[2024-11-04 12:45:05,053] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled Age (avg=-0.008197306847307117 std=0.9999098721983084) nulls=0
[2024-11-04 12:45:05,053] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FlushDFBuilder(num_flagonly)
[2024-11-04 12:45:05,053] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FlushDFBuilder(datetime_cyclical)
[2024-11-04 12:45:05,053] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FastSparseDummifyProcessor (Division Name)
[2024-11-04 12:45:05,054] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(4503, 5) nnz=4503
[2024-11-04 12:45:05,054] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FastSparseDummifyProcessor (Department Name)
[2024-11-04 12:45:05,056] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(4503, 8) nnz=4503
[2024-11-04 12:45:05,056] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FastSparseDummifyProcessor (Class Name)
[2024-11-04 12:45:05,057] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(4503, 22) nnz=4503
[2024-11-04 12:45:05,057] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:MultipleImputeMissingFromInput
[2024-11-04 12:45:05,057] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2024-11-04 12:45:05,057] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FlushDFBuilder(cat_flagpresence)
[2024-11-04 12:45:05,057] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:MultipleImputeMissingFromInput
[2024-11-04 12:45:05,057] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2024-11-04 12:45:05,057] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:MultipleImputeMissingFromInput
[2024-11-04 12:45:05,057] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2024-11-04 12:45:05,057] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FlushDFBuilder(interaction)
[2024-11-04 12:45:05,057] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RealignTarget
[2024-11-04 12:45:05,057] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] Realign target series = (4503,)
[2024-11-04 12:45:05,057] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] After realign target: (4503,)
[2024-11-04 12:45:05,057] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:DropRowsWhereNoTarget
[2024-11-04 12:45:05,057] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] Deleting 0 rows because one of ['target'] is missing or infinity
[2024-11-04 12:45:05,057] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] MF before = (4503, 39)
[2024-11-04 12:45:05,057] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] target before = (4503,)
[2024-11-04 12:45:05,058] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] MultiFrame, dropping rows: []
[2024-11-04 12:45:05,059] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] After DRWNT input_df=(4503, 10)
[2024-11-04 12:45:05,059] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] MF after = (4503, 39)
[2024-11-04 12:45:05,059] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] target after = (4503,)
[2024-11-04 12:45:05,059] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:DumpPipelineState
[2024-11-04 12:45:05,059] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] ********* Pipeline state (Before feature selection)
[2024-11-04 12:45:05,059] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]    input_df= (4503, 10) 
[2024-11-04 12:45:05,059] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]    current_mf=(4503, 39) 
[2024-11-04 12:45:05,059] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]    PPR: 
[2024-11-04 12:45:05,059] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]       target = <class 'pandas.core.series.Series'> ((4503,))
[2024-11-04 12:45:05,059] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:EmitCurrentMFAsResult
[2024-11-04 12:45:05,059] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 4503
[2024-11-04 12:45:05,059] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:DumpPipelineState
[2024-11-04 12:45:05,059] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] ********* Pipeline state (At end)
[2024-11-04 12:45:05,059] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]    input_df= (4503, 10) 
[2024-11-04 12:45:05,059] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]    current_mf=(0, 0) 
[2024-11-04 12:45:05,059] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]    PPR: 
[2024-11-04 12:45:05,059] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]       target = <class 'pandas.core.series.Series'> ((4503,))
[2024-11-04 12:45:05,059] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]       TRAIN = <class 'dataiku.doctor.multiframe.MultiFrame'> ((4503, 39))
[2024-11-04 12:45:05,059] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]       UNPROCESSED = <class 'pandas.core.frame.DataFrame'> ((4503, 10))
[2024-11-04 12:45:05,059] [14336/MainThread] [INFO] [root] test multiframe initial size: 4503
[2024-11-04 12:45:05,060] [14336/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Preprocessing test set
[2024-11-04 12:45:05,060] [14336/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Hyperparameter searching
[2024-11-04 12:45:05,060] [14336/MainThread] [INFO] [dataiku.doctor.sparse] prepare multiframe shape=(18138,39) est_filled_columns=7 fill_ratio=0.18
[2024-11-04 12:45:05,060] [14336/MainThread] [INFO] [dataiku.doctor.sparse] Sparse matrices support is explicitly disabled, using NPA
[2024-11-04 12:45:05,071] [14336/MainThread] [INFO] [dataiku.doctor.prediction.common] Create CLF from params: {'algorithm': 'RANDOM_FOREST_CLASSIFICATION', 'rf_classifier_grid': {'n_estimators': {'gridMode': 'EXPLICIT', 'randomMode': 'RANGE', 'limit': {'min': 1}, 'values': [100], 'range': {'min': 80, 'max': 200, 'nbValues': 3, 'scaling': 'LINEAR'}}, 'n_jobs': 4, 'max_tree_depth': {'gridMode': 'EXPLICIT', 'randomMode': 'RANGE', 'limit': {'min': 1}, 'values': [6, 13], 'range': {'min': 6, 'max': 20, 'nbValues': 3, 'scaling': 'LINEAR'}}, 'min_samples_leaf': {'gridMode': 'EXPLICIT', 'randomMode': 'RANGE', 'limit': {'min': 1}, 'values': [1], 'range': {'min': 1, 'max': 20, 'nbValues': 3, 'scaling': 'LINEAR'}}, 'selection_mode': 'sqrt', 'max_features': {'gridMode': 'EXPLICIT', 'randomMode': 'RANGE', 'limit': {'min': 1}, 'values': [5], 'range': {'min': 1, 'max': 20, 'nbValues': 3, 'scaling': 'LINEAR'}}, 'max_feature_prop': {'gridMode': 'EXPLICIT', 'randomMode': 'RANGE', 'limit': {'min': 1e-23, 'max': 1.0}, 'values': [0.3], 'range': {'min': 0.1, 'max': 0.7, 'nbValues': 3, 'scaling': 'LINEAR'}}, 'allow_sparse_matrices': False, 'enabled': True}, 'max_ensemble_nodes_serialized': 6000, 'metrics': {'customMetrics': [], 'evaluationMetric': 'ROC_AUC', 'costMatrixWeights': {'tpGain': 1.0, 'tnGain': 0.0, 'fpGain': -0.3, 'fnGain': 0.0}, 'liftPoint': 0.4, 'netUpliftPoint': 0.5, 'causalWeighting': 'NO_WEIGHTING'}, 'autoOptimizeThreshold': True, 'forcedClassifierThreshold': 0.0, 'gridLength': 2, 'grid_search_params': {'mode': 'KFOLD', 'splitRatio': 0.8, 'shuffleIterations': 1, 'nFolds': 5, 'cvSeed': 1337, 'stratified': True, 'grouped': False, 'strategy': 'GRID', 'bayesianOptimizer': 'SCIKIT_OPTIMIZE', 'randomized': True, 'seed': 1337, 'nIter': 0, 'nIterRandom': 5, 'timeout': 0, 'nJobs': 4, 'distributed': False, 'nContainers': 4, 'foldOffset': True, 'equalDurationFolds': False}, 'pluginAlgoCustomGridSearch': False, 'computeLearningCurves': False, 'skipExpensiveReports': False, 'propensityModeling': {'enabled': True, 'calibrateProbabilities': False, 'calibrationDataRatio': 0.1}} for algorithm RANDOM_FOREST_CLASSIFICATION
[2024-11-04 12:45:05,072] [14336/MainThread] [INFO] [dataiku.doctor.prediction.common] Using stratified K-Fold CV with k=5
[2024-11-04 12:45:05,076] [14336/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluation_monitor] No distributed container configuration is available to run this search
[2024-11-04 12:45:05,081] [14336/MainThread] [INFO] [dataiku.doctor.crossval.search_runner] Fitting 5 folds for each of 2 candidates, totalling 10 fits
[2024-11-04 12:45:05,081] [14336/MainThread] [INFO] [dataiku.base.utils] Detected ?/12 usable CPUs
[2024-11-04 12:45:05,081] [14336/MainThread] [INFO] [dataiku.doctor.crossval.search_runner] Execute hyperparameter search locally on 4 threads
[2024-11-04 12:45:05,082] [14336/MainThread] [INFO] [dataiku.doctor.crossval.strategies.grid_search_strategy] Running GridSearchStrategy for hyperparameters space: <dataiku.doctor.prediction.common.TreesHyperparametersSpace object at 0x7f25cb315b70>
[2024-11-04 12:45:05,082] [14336/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 0: {'max_depth': 6, 'max_features': 'sqrt', 'min_samples_leaf': 1, 'n_estimators': 100, 'min_samples_split': 3}
[2024-11-04 12:45:05,082] [14336/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.local_worker] Starting worker: local-0
[2024-11-04 12:45:05,082] [14336/MainThread] [INFO] [dataiku.doctor.distributed.work_scheduler] Enqueued task T1
[2024-11-04 12:45:05,082] [14336/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.local_worker] Started worker: local-0
[2024-11-04 12:45:05,082] [14336/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 1: {'max_depth': 6, 'max_features': 'sqrt', 'min_samples_leaf': 1, 'n_estimators': 100, 'min_samples_split': 3}
[2024-11-04 12:45:05,082] [14336/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task T1 on local-0...
[2024-11-04 12:45:05,082] [14336/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.local_worker] Starting worker: local-1
[2024-11-04 12:45:05,082] [14336/Thread-2:local-0] [INFO] [dataiku.doctor.crossval.search_context] Fit s=0: max_depth=6, max_features=sqrt, min_samples_leaf=1, n_estimators=100, min_samples_split=3 
[2024-11-04 12:45:05,082] [14336/MainThread] [INFO] [dataiku.doctor.distributed.work_scheduler] Enqueued task T2
[2024-11-04 12:45:05,082] [14336/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.local_worker] Started worker: local-1
[2024-11-04 12:45:05,082] [14336/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 2: {'max_depth': 6, 'max_features': 'sqrt', 'min_samples_leaf': 1, 'n_estimators': 100, 'min_samples_split': 3}
[2024-11-04 12:45:05,082] [14336/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task T2 on local-1...
[2024-11-04 12:45:05,083] [14336/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Fit s=1: max_depth=6, max_features=sqrt, min_samples_leaf=1, n_estimators=100, min_samples_split=3 
[2024-11-04 12:45:05,088] [14336/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.local_worker] Starting worker: local-2
[2024-11-04 12:45:05,088] [14336/MainThread] [INFO] [dataiku.doctor.distributed.work_scheduler] Enqueued task T3
[2024-11-04 12:45:05,088] [14336/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.local_worker] Started worker: local-2
[2024-11-04 12:45:05,089] [14336/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 3: {'max_depth': 6, 'max_features': 'sqrt', 'min_samples_leaf': 1, 'n_estimators': 100, 'min_samples_split': 3}
[2024-11-04 12:45:05,089] [14336/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task T3 on local-2...
[2024-11-04 12:45:05,092] [14336/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.local_worker] Starting worker: local-3
[2024-11-04 12:45:05,092] [14336/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Fit s=2: max_depth=6, max_features=sqrt, min_samples_leaf=1, n_estimators=100, min_samples_split=3 
[2024-11-04 12:45:05,092] [14336/MainThread] [INFO] [dataiku.doctor.distributed.work_scheduler] Enqueued task T4
[2024-11-04 12:45:05,092] [14336/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.local_worker] Started worker: local-3
[2024-11-04 12:45:05,093] [14336/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 4: {'max_depth': 6, 'max_features': 'sqrt', 'min_samples_leaf': 1, 'n_estimators': 100, 'min_samples_split': 3}
[2024-11-04 12:45:05,093] [14336/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task T4 on local-3...
[2024-11-04 12:45:05,093] [14336/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Fit s=3: max_depth=6, max_features=sqrt, min_samples_leaf=1, n_estimators=100, min_samples_split=3 
/data/home/samplemaster/dataiku-dss-13.2.0/python36.packages/sklearn/ensemble/base.py:154: UserWarning: Loky-backed parallel loops cannot be nested below threads, setting n_jobs=1
  n_jobs = min(effective_n_jobs(n_jobs), n_estimators)
/data/home/samplemaster/dataiku-dss-13.2.0/python36.packages/sklearn/ensemble/base.py:154: UserWarning: Loky-backed parallel loops cannot be nested below threads, setting n_jobs=1
  n_jobs = min(effective_n_jobs(n_jobs), n_estimators)
/data/home/samplemaster/dataiku-dss-13.2.0/python36.packages/sklearn/ensemble/base.py:154: UserWarning: Loky-backed parallel loops cannot be nested below threads, setting n_jobs=1
  n_jobs = min(effective_n_jobs(n_jobs), n_estimators)
/data/home/samplemaster/dataiku-dss-13.2.0/python36.packages/sklearn/ensemble/base.py:154: UserWarning: Loky-backed parallel loops cannot be nested below threads, setting n_jobs=1
  n_jobs = min(effective_n_jobs(n_jobs), n_estimators)
/data/home/samplemaster/dataiku-dss-13.2.0/python36.packages/sklearn/ensemble/base.py:154: UserWarning: Loky-backed parallel loops cannot be nested below threads, setting n_jobs=1
  n_jobs = min(effective_n_jobs(n_jobs), n_estimators)
/data/home/samplemaster/dataiku-dss-13.2.0/python36.packages/sklearn/ensemble/base.py:154: UserWarning: Loky-backed parallel loops cannot be nested below threads, setting n_jobs=1
  n_jobs = min(effective_n_jobs(n_jobs), n_estimators)
/data/home/samplemaster/dataiku-dss-13.2.0/python36.packages/sklearn/ensemble/base.py:154: UserWarning: Loky-backed parallel loops cannot be nested below threads, setting n_jobs=1
  n_jobs = min(effective_n_jobs(n_jobs), n_estimators)
/data/home/samplemaster/dataiku-dss-13.2.0/python36.packages/sklearn/ensemble/base.py:154: UserWarning: Loky-backed parallel loops cannot be nested below threads, setting n_jobs=1
  n_jobs = min(effective_n_jobs(n_jobs), n_estimators)
[2024-11-04 12:45:06,548] [14336/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Done s=2: max_depth=6, max_features=sqrt, min_samples_leaf=1, n_estimators=100, min_samples_split=3 (ft=1.1 st=0.3 sc=0.7510383419705828, sg=1)
[2024-11-04 12:45:06,548] [14336/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Task T3 done on local-2
[2024-11-04 12:45:06,548] [14336/MainThread] [INFO] [dataiku.doctor.distributed.work_scheduler] Enqueued task T5
[2024-11-04 12:45:06,549] [14336/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task T5 on local-2...
[2024-11-04 12:45:06,549] [14336/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Fit s=4: max_depth=6, max_features=sqrt, min_samples_leaf=1, n_estimators=100, min_samples_split=3 
[2024-11-04 12:45:06,549] [14336/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 0: {'max_depth': 13, 'max_features': 'sqrt', 'min_samples_leaf': 1, 'n_estimators': 100, 'min_samples_split': 3}
[2024-11-04 12:45:06,626] [14336/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Done s=1: max_depth=6, max_features=sqrt, min_samples_leaf=1, n_estimators=100, min_samples_split=3 (ft=1.2 st=0.4 sc=0.7380824281484193, sg=1)
[2024-11-04 12:45:06,626] [14336/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Task T2 done on local-1
[2024-11-04 12:45:06,627] [14336/MainThread] [INFO] [dataiku.doctor.distributed.work_scheduler] Enqueued task T6
[2024-11-04 12:45:06,627] [14336/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 1: {'max_depth': 13, 'max_features': 'sqrt', 'min_samples_leaf': 1, 'n_estimators': 100, 'min_samples_split': 3}
[2024-11-04 12:45:06,627] [14336/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task T6 on local-1...
[2024-11-04 12:45:06,628] [14336/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Fit s=0: max_depth=13, max_features=sqrt, min_samples_leaf=1, n_estimators=100, min_samples_split=3 
[2024-11-04 12:45:06,753] [14336/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Done s=3: max_depth=6, max_features=sqrt, min_samples_leaf=1, n_estimators=100, min_samples_split=3 (ft=1.2 st=0.5 sc=0.7504717568889585, sg=1)
[2024-11-04 12:45:06,753] [14336/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Task T4 done on local-3
[2024-11-04 12:45:06,753] [14336/MainThread] [INFO] [dataiku.doctor.distributed.work_scheduler] Enqueued task T7
[2024-11-04 12:45:06,753] [14336/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 2: {'max_depth': 13, 'max_features': 'sqrt', 'min_samples_leaf': 1, 'n_estimators': 100, 'min_samples_split': 3}
[2024-11-04 12:45:06,754] [14336/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task T7 on local-3...
[2024-11-04 12:45:06,754] [14336/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Fit s=1: max_depth=13, max_features=sqrt, min_samples_leaf=1, n_estimators=100, min_samples_split=3 
[2024-11-04 12:45:07,040] [14336/Thread-2:local-0] [INFO] [dataiku.doctor.crossval.search_context] Done s=0: max_depth=6, max_features=sqrt, min_samples_leaf=1, n_estimators=100, min_samples_split=3 (ft=1.2 st=0.8 sc=0.7472317813323541, sg=1)
[2024-11-04 12:45:07,041] [14336/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.work_scheduler] Task T1 done on local-0
[2024-11-04 12:45:07,041] [14336/MainThread] [INFO] [dataiku.doctor.distributed.work_scheduler] Enqueued task T8
[2024-11-04 12:45:07,043] [14336/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 3: {'max_depth': 13, 'max_features': 'sqrt', 'min_samples_leaf': 1, 'n_estimators': 100, 'min_samples_split': 3}
[2024-11-04 12:45:07,043] [14336/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task T8 on local-0...
[2024-11-04 12:45:07,043] [14336/Thread-2:local-0] [INFO] [dataiku.doctor.crossval.search_context] Fit s=2: max_depth=13, max_features=sqrt, min_samples_leaf=1, n_estimators=100, min_samples_split=3 
/data/home/samplemaster/dataiku-dss-13.2.0/python36.packages/sklearn/ensemble/base.py:154: UserWarning: Loky-backed parallel loops cannot be nested below threads, setting n_jobs=1
  n_jobs = min(effective_n_jobs(n_jobs), n_estimators)
/data/home/samplemaster/dataiku-dss-13.2.0/python36.packages/sklearn/ensemble/base.py:154: UserWarning: Loky-backed parallel loops cannot be nested below threads, setting n_jobs=1
  n_jobs = min(effective_n_jobs(n_jobs), n_estimators)
/data/home/samplemaster/dataiku-dss-13.2.0/python36.packages/sklearn/ensemble/base.py:154: UserWarning: Loky-backed parallel loops cannot be nested below threads, setting n_jobs=1
  n_jobs = min(effective_n_jobs(n_jobs), n_estimators)
/data/home/samplemaster/dataiku-dss-13.2.0/python36.packages/sklearn/ensemble/base.py:154: UserWarning: Loky-backed parallel loops cannot be nested below threads, setting n_jobs=1
  n_jobs = min(effective_n_jobs(n_jobs), n_estimators)
[2024-11-04 12:45:07,695] [14336/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Done s=4: max_depth=6, max_features=sqrt, min_samples_leaf=1, n_estimators=100, min_samples_split=3 (ft=0.6 st=0.6 sc=0.754560869311542, sg=1)
[2024-11-04 12:45:07,695] [14336/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Task T5 done on local-2
[2024-11-04 12:45:07,695] [14336/MainThread] [INFO] [dataiku.doctor.distributed.work_scheduler] Enqueued task T9
[2024-11-04 12:45:07,695] [14336/MainThread] [INFO] [dataiku.doctor.crossval.search_evaluator] Schedule evaluation on split 4: {'max_depth': 13, 'max_features': 'sqrt', 'min_samples_leaf': 1, 'n_estimators': 100, 'min_samples_split': 3}
[2024-11-04 12:45:07,696] [14336/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task T9 on local-2...
[2024-11-04 12:45:07,696] [14336/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Fit s=3: max_depth=13, max_features=sqrt, min_samples_leaf=1, n_estimators=100, min_samples_split=3 
/data/home/samplemaster/dataiku-dss-13.2.0/python36.packages/sklearn/ensemble/base.py:154: UserWarning: Loky-backed parallel loops cannot be nested below threads, setting n_jobs=1
  n_jobs = min(effective_n_jobs(n_jobs), n_estimators)
/data/home/samplemaster/dataiku-dss-13.2.0/python36.packages/sklearn/ensemble/base.py:154: UserWarning: Loky-backed parallel loops cannot be nested below threads, setting n_jobs=1
  n_jobs = min(effective_n_jobs(n_jobs), n_estimators)
/data/home/samplemaster/dataiku-dss-13.2.0/python36.packages/sklearn/ensemble/base.py:154: UserWarning: Loky-backed parallel loops cannot be nested below threads, setting n_jobs=1
  n_jobs = min(effective_n_jobs(n_jobs), n_estimators)
[2024-11-04 12:45:08,011] [14336/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Done s=0: max_depth=13, max_features=sqrt, min_samples_leaf=1, n_estimators=100, min_samples_split=3 (ft=0.8 st=0.5 sc=0.7468969373344927, sg=1)
[2024-11-04 12:45:08,011] [14336/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Task T6 done on local-1
[2024-11-04 12:45:08,012] [14336/MainThread] [INFO] [dataiku.doctor.distributed.work_scheduler] Enqueued task T10
[2024-11-04 12:45:08,012] [14336/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Running task T10 on local-1...
[2024-11-04 12:45:08,012] [14336/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Fit s=4: max_depth=13, max_features=sqrt, min_samples_leaf=1, n_estimators=100, min_samples_split=3 
/data/home/samplemaster/dataiku-dss-13.2.0/python36.packages/sklearn/ensemble/base.py:154: UserWarning: Loky-backed parallel loops cannot be nested below threads, setting n_jobs=1
  n_jobs = min(effective_n_jobs(n_jobs), n_estimators)
[2024-11-04 12:45:08,115] [14336/Thread-5:local-3] [INFO] [dataiku.doctor.crossval.search_context] Done s=1: max_depth=13, max_features=sqrt, min_samples_leaf=1, n_estimators=100, min_samples_split=3 (ft=0.8 st=0.5 sc=0.7329531642543636, sg=1)
[2024-11-04 12:45:08,115] [14336/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.work_scheduler] Task T7 done on local-3
/data/home/samplemaster/dataiku-dss-13.2.0/python36.packages/sklearn/ensemble/base.py:154: UserWarning: Loky-backed parallel loops cannot be nested below threads, setting n_jobs=1
  n_jobs = min(effective_n_jobs(n_jobs), n_estimators)
[2024-11-04 12:45:08,242] [14336/Thread-2:local-0] [INFO] [dataiku.doctor.crossval.search_context] Done s=2: max_depth=13, max_features=sqrt, min_samples_leaf=1, n_estimators=100, min_samples_split=3 (ft=0.7 st=0.5 sc=0.7471344866996434, sg=1)
[2024-11-04 12:45:08,242] [14336/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.work_scheduler] Task T8 done on local-0
/data/home/samplemaster/dataiku-dss-13.2.0/python36.packages/sklearn/ensemble/base.py:154: UserWarning: Loky-backed parallel loops cannot be nested below threads, setting n_jobs=1
  n_jobs = min(effective_n_jobs(n_jobs), n_estimators)
/data/home/samplemaster/dataiku-dss-13.2.0/python36.packages/sklearn/ensemble/base.py:154: UserWarning: Loky-backed parallel loops cannot be nested below threads, setting n_jobs=1
  n_jobs = min(effective_n_jobs(n_jobs), n_estimators)
[2024-11-04 12:45:08,416] [14336/Thread-4:local-2] [INFO] [dataiku.doctor.crossval.search_context] Done s=3: max_depth=13, max_features=sqrt, min_samples_leaf=1, n_estimators=100, min_samples_split=3 (ft=0.4 st=0.4 sc=0.7466276750097431, sg=1)
[2024-11-04 12:45:08,416] [14336/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.work_scheduler] Task T9 done on local-2
/data/home/samplemaster/dataiku-dss-13.2.0/python36.packages/sklearn/ensemble/base.py:154: UserWarning: Loky-backed parallel loops cannot be nested below threads, setting n_jobs=1
  n_jobs = min(effective_n_jobs(n_jobs), n_estimators)
[2024-11-04 12:45:08,653] [14336/Thread-3:local-1] [INFO] [dataiku.doctor.crossval.search_context] Done s=4: max_depth=13, max_features=sqrt, min_samples_leaf=1, n_estimators=100, min_samples_split=3 (ft=0.3 st=0.3 sc=0.7586696368648993, sg=1)
[2024-11-04 12:45:08,653] [14336/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Task T10 done on local-1
[2024-11-04 12:45:09,094] [14336/Thread-1] [INFO] [dataiku.doctor.crossval.search_runner] Completed search for hyperparameters
[2024-11-04 12:45:09,094] [14336/Thread-1] [INFO] [dataiku.doctor.distributed.work_scheduler] Scheduler has been soft interrupted
[2024-11-04 12:45:09,095] [14336/MainThread] [INFO] [dataiku.doctor.distributed.work_scheduler] Scheduler has been hard interrupted (shutdown)
[2024-11-04 12:45:09,095] [14336/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.local_worker] Stopping worker: local-3
[2024-11-04 12:45:09,095] [14336/Thread-5:local-3] [INFO] [dataiku.doctor.distributed.local_worker] Stopped worker: local-3
[2024-11-04 12:45:09,095] [14336/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.local_worker] Stopping worker: local-0
[2024-11-04 12:45:09,095] [14336/Thread-2:local-0] [INFO] [dataiku.doctor.distributed.local_worker] Stopped worker: local-0
[2024-11-04 12:45:09,095] [14336/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.local_worker] Stopping worker: local-2
[2024-11-04 12:45:09,095] [14336/Thread-4:local-2] [INFO] [dataiku.doctor.distributed.local_worker] Stopped worker: local-2
[2024-11-04 12:45:09,095] [14336/Thread-3:local-1] [ERROR] [dataiku.doctor.distributed.work_scheduler] All workers are dead, interrupt the scheduler
[2024-11-04 12:45:09,095] [14336/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.local_worker] Stopping worker: local-1
[2024-11-04 12:45:09,095] [14336/Thread-3:local-1] [INFO] [dataiku.doctor.distributed.local_worker] Stopped worker: local-1
[2024-11-04 12:45:09,095] [14336/MainThread] [INFO] [dataiku.doctor.crossval.search_runner] Hyperparameter search done, best_parameters being : {'max_depth': 6, 'max_features': 'sqrt', 'min_samples_leaf': 1, 'n_estimators': 100, 'min_samples_split': 3}
[2024-11-04 12:45:09,096] [14336/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Hyperparameter searching
[2024-11-04 12:45:09,096] [14336/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Fitting model
[2024-11-04 12:45:09,096] [14336/MainThread] [INFO] [dataiku.doctor.prediction.common] Fitting model:
[2024-11-04 12:45:09,096] [14336/MainThread] [INFO] [dataiku.doctor.prediction.common]   Model is: RandomForestClassifier(bootstrap=True,
            class_weight={0: 0.36009529481834424, 1: 0.9225839267548321, 2: 1.6194642857142858, 3: 2.9661488143908423, 4: 5.422421524663677},
            criterion='gini', max_depth=6, max_features='sqrt',
            max_leaf_nodes=None, min_impurity_decrease=0.0,
            min_impurity_split=None, min_samples_leaf=1,
            min_samples_split=3, min_weight_fraction_leaf=0.0,
            n_estimators=100, n_jobs=4, oob_score=False, random_state=1337,
            verbose=0, warm_start=False)
[2024-11-04 12:45:09,096] [14336/MainThread] [INFO] [dataiku.doctor.prediction.common]   train_X class: <class 'numpy.ndarray'>
[2024-11-04 12:45:09,096] [14336/MainThread] [INFO] [dataiku.doctor.prediction.common]   train_X shape: (18138, 39)
[2024-11-04 12:45:09,096] [14336/MainThread] [INFO] [dataiku.doctor.prediction.common]   train_y shape: (18138,)
[2024-11-04 12:45:09,422] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_fit] Obtained RF CLF params: {'bootstrap': True, 'class_weight': {0: 0.36009529481834424, 1: 0.9225839267548321, 2: 1.6194642857142858, 3: 2.9661488143908423, 4: 5.422421524663677}, 'criterion': 'gini', 'max_depth': 6, 'max_features': 'sqrt', 'max_leaf_nodes': None, 'min_impurity_decrease': 0.0, 'min_impurity_split': None, 'min_samples_leaf': 1, 'min_samples_split': 3, 'min_weight_fraction_leaf': 0.0, 'n_estimators': 100, 'n_jobs': 4, 'oob_score': False, 'random_state': 1337, 'verbose': 0, 'warm_start': False} 
[2024-11-04 12:45:09,422] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_fit] Output params are {'resolved': {'algorithm': 'RANDOM_FOREST_CLASSIFICATION', 'skipExpensiveReports': False, 'rf': {'estimators': 100, 'max_tree_depth': 6, 'min_samples_leaf': 1, 'selection_mode': 'sqrt'}}, 'other': {'rf_min_samples_split': 3}}
[2024-11-04 12:45:09,626] [14336/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Fitting model
[2024-11-04 12:45:09,626] [14336/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Saving model
[2024-11-04 12:45:09,658] [14336/MainThread] [INFO] [dataiku.base.dku_pickle] Setting cloudpickle as the pickle dumping tool
[2024-11-04 12:45:09,928] [14336/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Saving model
[2024-11-04 12:45:09,929] [14336/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Scoring model
[2024-11-04 12:45:09,929] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] Intrinsic scoring
[2024-11-04 12:45:09,929] [14336/MainThread] [INFO] [dataiku.doctor.prediction.scoring_base] Computing feature importance
[2024-11-04 12:45:10,131] [14336/MainThread] [INFO] [dataiku.doctor.prediction.linear_coefficients_computation] Not computing linear coefficients because attribute `coef_` is not present
[2024-11-04 12:45:10,131] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] Extracting rescalers
[2024-11-04 12:45:10,131] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] Creating random forest trees summary
[2024-11-04 12:45:10,335] [14336/MainThread] [INFO] [dataiku.doctor.sparse] prepare multiframe shape=(4503,39) est_filled_columns=7 fill_ratio=0.18
[2024-11-04 12:45:10,335] [14336/MainThread] [INFO] [dataiku.doctor.sparse] Sparse matrices support is explicitly disabled, using NPA
[2024-11-04 12:45:10,538] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] Will use probas : True
[2024-11-04 12:45:10,540] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] Calculated confusion matrix
/data/home/samplemaster/dataiku-dss-13.2.0/python36.packages/sklearn/metrics/classification.py:1143: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.
  'precision', 'predicted', average, warn_for)
/data/home/samplemaster/dataiku-dss-13.2.0/python/dataiku/doctor/utils/calibration.py:59: RuntimeWarning: invalid value encountered in true_divide
  freq_pos = y_true[i_start:i_end].sum() / weight
/data/home/samplemaster/dataiku-dss-13.2.0/python/dataiku/doctor/utils/calibration.py:60: RuntimeWarning: invalid value encountered in double_scalars
  avg_pred = y_prob[i_start:i_end].sum() / weight
[2024-11-04 12:45:10,602] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] Density for 5 (id 0)
[2024-11-04 12:45:10,602] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] valid_y shape = (4503,)
[2024-11-04 12:45:10,602] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] CP: <class 'numpy.ndarray'> ((4503,))
[2024-11-04 12:45:10,602] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] Actual shape = (4503,)
[2024-11-04 12:45:10,602] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] MASK is <class 'numpy.ndarray'> 
[2024-11-04 12:45:10,602] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] Class proba shape (2466,)
[2024-11-04 12:45:10,663] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] Density for 4 (id 1)
[2024-11-04 12:45:10,663] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] valid_y shape = (4503,)
[2024-11-04 12:45:10,663] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] CP: <class 'numpy.ndarray'> ((4503,))
[2024-11-04 12:45:10,663] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] Actual shape = (4503,)
[2024-11-04 12:45:10,663] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] MASK is <class 'numpy.ndarray'> 
[2024-11-04 12:45:10,663] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] Class proba shape (976,)
[2024-11-04 12:45:10,716] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] Density for 3 (id 2)
[2024-11-04 12:45:10,716] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] valid_y shape = (4503,)
[2024-11-04 12:45:10,716] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] CP: <class 'numpy.ndarray'> ((4503,))
[2024-11-04 12:45:10,716] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] Actual shape = (4503,)
[2024-11-04 12:45:10,716] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] MASK is <class 'numpy.ndarray'> 
[2024-11-04 12:45:10,716] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] Class proba shape (583,)
[2024-11-04 12:45:10,737] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] Density for 2 (id 3)
[2024-11-04 12:45:10,737] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] valid_y shape = (4503,)
[2024-11-04 12:45:10,737] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] CP: <class 'numpy.ndarray'> ((4503,))
[2024-11-04 12:45:10,737] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] Actual shape = (4503,)
[2024-11-04 12:45:10,737] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] MASK is <class 'numpy.ndarray'> 
[2024-11-04 12:45:10,737] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] Class proba shape (326,)
[2024-11-04 12:45:10,782] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] Density for 1 (id 4)
[2024-11-04 12:45:10,782] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] valid_y shape = (4503,)
[2024-11-04 12:45:10,782] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] CP: <class 'numpy.ndarray'> ((4503,))
[2024-11-04 12:45:10,782] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] Actual shape = (4503,)
[2024-11-04 12:45:10,782] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] MASK is <class 'numpy.ndarray'> 
[2024-11-04 12:45:10,782] [14336/MainThread] [INFO] [dataiku.doctor.prediction.classification_scoring] Class proba shape (152,)
[2024-11-04 12:45:11,051] [14336/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Scoring model
[2024-11-04 12:45:11,051] [14336/MainThread] [INFO] [dataiku.doctor.utils.listener] START -  Preparing explainability
[2024-11-04 12:45:11,051] [14336/MainThread] [INFO] [dataiku.doctor.prediction_entrypoints] Starting to compute global explanations
[2024-11-04 12:45:11,051] [14336/MainThread] [INFO] [dataiku.doctor.prediction_entrypoints] Building model handler
[2024-11-04 12:45:11,052] [14336/MainThread] [INFO] [dataiku.doctor.utils.gpu_execution] Not logging `nvidia-smi`: `useGPU` is False
[2024-11-04 12:45:11,056] [14336/MainThread] [INFO] [dataiku.doctor.preprocessing_handler] generating interactions
[2024-11-04 12:45:11,056] [14336/MainThread] [INFO] [dataiku.doctor.preprocessing_handler] {'feature_selection_params': {'method': 'NONE', 'random_forest_params': {'n_trees': 30, 'depth': 10, 'n_features': 25}, 'lasso_params': {'alpha': [0.01, 0.1, 1.0, 10.0, 100.0], 'cross_validate': True}, 'pca_params': {'n_features': 25, 'variance_proportion': 0.9}, 'correlation_params': {'min_abs_correlation': 0.0, 'max_abs_correlation': 1.0, 'n_features': 25}, 'custom_params': {'code': '# type your code here'}}, 'preprocessingFitSampleRatio': 1.0, 'preprocessingFitSampleSeed': 1337, 'target_remapping': [{'sourceValue': '5', 'mappedValue': 0, 'sampleFreq': 5527}, {'sourceValue': '4', 'mappedValue': 1, 'sampleFreq': 2184}, {'sourceValue': '3', 'mappedValue': 2, 'sampleFreq': 1248}, {'sourceValue': '2', 'mappedValue': 3, 'sampleFreq': 682}, {'sourceValue': '1', 'mappedValue': 4, 'sampleFreq': 359}], 'skipPreprocessing': False, 'per_feature': {'Recommended IND': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'keep_regular': False, 'rescaling': 'AVGSTD', 'monotonic': 'NONE', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'datetime_cyclical_periods': [], 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Division Name': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 100, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'NONE', 'impact_method': 'M_ESTIMATOR', 'impact_m': 10, 'impact_kfold': True, 'impact_kfold_k': 5, 'impact_kfold_seed': 1337, 'ordinal_order': 'COUNT', 'ordinal_ascending': False, 'ordinal_default_mode': 'HIGHEST', 'ordinal_default_value': 0, 'frequency_default_mode': 'EXPLICIT', 'frequency_default_value': 0.0, 'frequency_normalized': True, 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Text'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Department Name': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 100, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'NONE', 'impact_method': 'M_ESTIMATOR', 'impact_m': 10, 'impact_kfold': True, 'impact_kfold_k': 5, 'impact_kfold_seed': 1337, 'ordinal_order': 'COUNT', 'ordinal_ascending': False, 'ordinal_default_mode': 'HIGHEST', 'ordinal_default_value': 0, 'frequency_default_mode': 'EXPLICIT', 'frequency_default_value': 0.0, 'frequency_normalized': True, 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Text'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Positive Feedback Count': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'keep_regular': False, 'rescaling': 'AVGSTD', 'monotonic': 'NONE', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'datetime_cyclical_periods': [], 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Review Text': {'text_handling': 'TOKENIZE_HASHING_SVD', 'minRowsRatio': 0.001, 'maxRowsRatio': 0.8, 'maxWords': 0, 'ngramMinSize': 1, 'ngramMaxSize': 1, 'hashSize': 200000, 'hashSVDSVDLimit': 50000, 'hashSVDSVDComponents': 100, 'stopWordsMode': 'NONE', 'maxSequenceLength': 128, 'sentenceEmbeddingBatchSize': 32, 'useCustomVectorizer': False, 'isStructuredRef': False, 'name': 'Review Text', 'role': 'REJECT', 'type': 'TEXT', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'FreeText'}, 'autoReason': 'REJECT_DEFAULT_TEXT_HANDLING', 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Rating': {'generate_derivative': False, 'impute_constant_value': 0.0, 'keep_regular': False, 'monotonic': 'NONE', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'datetime_cyclical_periods': [], 'role': 'TARGET', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Class Name': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 100, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'NONE', 'impact_method': 'M_ESTIMATOR', 'impact_m': 10, 'impact_kfold': True, 'impact_kfold_k': 5, 'impact_kfold_seed': 1337, 'ordinal_order': 'COUNT', 'ordinal_ascending': False, 'ordinal_default_mode': 'HIGHEST', 'ordinal_default_value': 0, 'frequency_default_mode': 'EXPLICIT', 'frequency_default_value': 0.0, 'frequency_normalized': True, 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Text'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Title': {'text_handling': 'TOKENIZE_HASHING_SVD', 'minRowsRatio': 0.001, 'maxRowsRatio': 0.8, 'maxWords': 0, 'ngramMinSize': 1, 'ngramMaxSize': 1, 'hashSize': 200000, 'hashSVDSVDLimit': 50000, 'hashSVDSVDComponents': 100, 'stopWordsMode': 'NONE', 'maxSequenceLength': 128, 'sentenceEmbeddingBatchSize': 32, 'useCustomVectorizer': False, 'isStructuredRef': False, 'name': 'Title', 'role': 'REJECT', 'type': 'TEXT', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'FreeText'}, 'autoReason': 'REJECT_DEFAULT_TEXT_HANDLING', 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Clothing ID': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'keep_regular': False, 'rescaling': 'AVGSTD', 'monotonic': 'NONE', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'datetime_cyclical_periods': [], 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Age': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'keep_regular': False, 'rescaling': 'AVGSTD', 'monotonic': 'NONE', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'datetime_cyclical_periods': [], 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}}, 'reduce': {'enabled': False, 'kept_variance': 0.0}, 'feature_generation': {'pairwise_linear': {'behavior': 'DISABLED'}, 'polynomial_combinations': {'behavior': 'DISABLED'}, 'manual_interactions': {'interactions': []}, 'numericals_clustering': {'k': 0, 'all_features': False, 'input_features': [], 'behavior': 'DISABLED'}, 'categoricals_count_transformer': {'all_features': False, 'input_features': [], 'behavior': 'DISABLED'}}}
[2024-11-04 12:45:11,056] [14336/MainThread] [INFO] [dataiku.doctor.preprocessing_handler] No feature selection to perform
[2024-11-04 12:45:11,056] [14336/MainThread] [INFO] [dataiku.doctor.preprocessing_handler] generating interactions
[2024-11-04 12:45:11,056] [14336/MainThread] [INFO] [dataiku.doctor.preprocessing_handler] {'feature_selection_params': {'method': 'NONE', 'random_forest_params': {'n_trees': 30, 'depth': 10, 'n_features': 25}, 'lasso_params': {'alpha': [0.01, 0.1, 1.0, 10.0, 100.0], 'cross_validate': True}, 'pca_params': {'n_features': 25, 'variance_proportion': 0.9}, 'correlation_params': {'min_abs_correlation': 0.0, 'max_abs_correlation': 1.0, 'n_features': 25}, 'custom_params': {'code': '# type your code here'}}, 'preprocessingFitSampleRatio': 1.0, 'preprocessingFitSampleSeed': 1337, 'target_remapping': [{'sourceValue': '5', 'mappedValue': 0, 'sampleFreq': 5527}, {'sourceValue': '4', 'mappedValue': 1, 'sampleFreq': 2184}, {'sourceValue': '3', 'mappedValue': 2, 'sampleFreq': 1248}, {'sourceValue': '2', 'mappedValue': 3, 'sampleFreq': 682}, {'sourceValue': '1', 'mappedValue': 4, 'sampleFreq': 359}], 'skipPreprocessing': False, 'per_feature': {'Recommended IND': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'keep_regular': False, 'rescaling': 'AVGSTD', 'monotonic': 'NONE', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'datetime_cyclical_periods': [], 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Division Name': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 100, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'NONE', 'impact_method': 'M_ESTIMATOR', 'impact_m': 10, 'impact_kfold': True, 'impact_kfold_k': 5, 'impact_kfold_seed': 1337, 'ordinal_order': 'COUNT', 'ordinal_ascending': False, 'ordinal_default_mode': 'HIGHEST', 'ordinal_default_value': 0, 'frequency_default_mode': 'EXPLICIT', 'frequency_default_value': 0.0, 'frequency_normalized': True, 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Text'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Department Name': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 100, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'NONE', 'impact_method': 'M_ESTIMATOR', 'impact_m': 10, 'impact_kfold': True, 'impact_kfold_k': 5, 'impact_kfold_seed': 1337, 'ordinal_order': 'COUNT', 'ordinal_ascending': False, 'ordinal_default_mode': 'HIGHEST', 'ordinal_default_value': 0, 'frequency_default_mode': 'EXPLICIT', 'frequency_default_value': 0.0, 'frequency_normalized': True, 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Text'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Positive Feedback Count': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'keep_regular': False, 'rescaling': 'AVGSTD', 'monotonic': 'NONE', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'datetime_cyclical_periods': [], 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Review Text': {'text_handling': 'TOKENIZE_HASHING_SVD', 'minRowsRatio': 0.001, 'maxRowsRatio': 0.8, 'maxWords': 0, 'ngramMinSize': 1, 'ngramMaxSize': 1, 'hashSize': 200000, 'hashSVDSVDLimit': 50000, 'hashSVDSVDComponents': 100, 'stopWordsMode': 'NONE', 'maxSequenceLength': 128, 'sentenceEmbeddingBatchSize': 32, 'useCustomVectorizer': False, 'isStructuredRef': False, 'name': 'Review Text', 'role': 'REJECT', 'type': 'TEXT', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'FreeText'}, 'autoReason': 'REJECT_DEFAULT_TEXT_HANDLING', 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Rating': {'generate_derivative': False, 'impute_constant_value': 0.0, 'keep_regular': False, 'monotonic': 'NONE', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'datetime_cyclical_periods': [], 'role': 'TARGET', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Class Name': {'category_handling': 'DUMMIFY', 'missing_handling': 'NONE', 'missing_impute_with': 'MODE', 'dummy_clip': 'MAX_NB_CATEGORIES', 'cumulative_proportion': 0.95, 'min_samples': 10, 'max_nb_categories': 100, 'max_cat_safety': 200, 'nb_bins_hashing': 1048576, 'hash_whole_categories': True, 'dummy_drop': 'NONE', 'impact_method': 'M_ESTIMATOR', 'impact_m': 10, 'impact_kfold': True, 'impact_kfold_k': 5, 'impact_kfold_seed': 1337, 'ordinal_order': 'COUNT', 'ordinal_ascending': False, 'ordinal_default_mode': 'HIGHEST', 'ordinal_default_value': 0, 'frequency_default_mode': 'EXPLICIT', 'frequency_default_value': 0.0, 'frequency_normalized': True, 'role': 'INPUT', 'type': 'CATEGORY', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'Text'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Title': {'text_handling': 'TOKENIZE_HASHING_SVD', 'minRowsRatio': 0.001, 'maxRowsRatio': 0.8, 'maxWords': 0, 'ngramMinSize': 1, 'ngramMaxSize': 1, 'hashSize': 200000, 'hashSVDSVDLimit': 50000, 'hashSVDSVDComponents': 100, 'stopWordsMode': 'NONE', 'maxSequenceLength': 128, 'sentenceEmbeddingBatchSize': 32, 'useCustomVectorizer': False, 'isStructuredRef': False, 'name': 'Title', 'role': 'REJECT', 'type': 'TEXT', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'FreeText'}, 'autoReason': 'REJECT_DEFAULT_TEXT_HANDLING', 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Clothing ID': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'keep_regular': False, 'rescaling': 'AVGSTD', 'monotonic': 'NONE', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'datetime_cyclical_periods': [], 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}, 'Age': {'generate_derivative': False, 'numerical_handling': 'REGULAR', 'missing_handling': 'IMPUTE', 'missing_impute_with': 'MEAN', 'impute_constant_value': 0.0, 'keep_regular': False, 'rescaling': 'AVGSTD', 'monotonic': 'NONE', 'quantile_bin_nb_bins': 4, 'binarize_threshold_mode': 'MEDIAN', 'binarize_constant_threshold': 0.0, 'datetime_cyclical_periods': [], 'role': 'INPUT', 'type': 'NUMERIC', 'state': {'userModified': False, 'autoModifiedByDSS': False, 'recordedMeaning': 'LongMeaning'}, 'customHandlingCode': '', 'customProcessorWantsMatrix': False, 'sendToInput': 'main'}}, 'reduce': {'enabled': False, 'kept_variance': 0.0}, 'feature_generation': {'pairwise_linear': {'behavior': 'DISABLED'}, 'polynomial_combinations': {'behavior': 'DISABLED'}, 'manual_interactions': {'interactions': []}, 'numericals_clustering': {'k': 0, 'all_features': False, 'input_features': [], 'behavior': 'DISABLED'}, 'categoricals_count_transformer': {'all_features': False, 'input_features': [], 'behavior': 'DISABLED'}}}
[2024-11-04 12:45:11,056] [14336/MainThread] [INFO] [dataiku.doctor.preprocessing_handler] No feature selection to perform
[2024-11-04 12:45:11,056] [14336/MainThread] [INFO] [dataiku.doctor.individual_explainer] Preparing explainer components: column importance, background rows and feature histograms
[2024-11-04 12:45:11,056] [14336/MainThread] [INFO] [root] Reading with dtypes: None
[2024-11-04 12:45:11,056] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Clothing ID: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2024-11-04 12:45:11,056] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Age: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2024-11-04 12:45:11,056] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Title: None (schema_type=string feature_type=TEXT feature_role=REJECT)
[2024-11-04 12:45:11,056] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Review Text: None (schema_type=string feature_type=TEXT feature_role=REJECT)
[2024-11-04 12:45:11,056] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Rating: <class 'numpy.object_'> (schema_type=bigint feature_type=NUMERIC feature_role=TARGET)
[2024-11-04 12:45:11,056] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Recommended IND: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2024-11-04 12:45:11,056] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Positive Feedback Count: <class 'numpy.float64'> (schema_type=bigint feature_type=NUMERIC feature_role=INPUT)
[2024-11-04 12:45:11,056] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Division Name: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2024-11-04 12:45:11,056] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Department Name: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2024-11-04 12:45:11,056] [14336/MainThread] [INFO] [dataiku.doctor.utils] Computed dtype for Class Name: str (schema_type=string feature_type=CATEGORY feature_role=INPUT)
[2024-11-04 12:45:11,056] [14336/MainThread] [INFO] [root] Reading with FIXED dtypes: {'Clothing ID': <class 'numpy.float64'>, 'Age': <class 'numpy.float64'>, 'Rating': <class 'numpy.object_'>, 'Recommended IND': <class 'numpy.float64'>, 'Positive Feedback Count': <class 'numpy.float64'>, 'Division Name': 'str', 'Department Name': 'str', 'Class Name': 'str'}
[2024-11-04 12:45:11,135] [14336/MainThread] [INFO] [root] Loaded table
[2024-11-04 12:45:11,135] [14336/MainThread] [INFO] [dataiku.doctor.utils] Coercion done
[2024-11-04 12:45:11,135] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 18138
[2024-11-04 12:45:11,135] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:MultipleImputeMissingFromInput
[2024-11-04 12:45:11,136] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {'Recommended IND': 0.8202668430918514, 'Positive Feedback Count': 2.6301135737126473, 'Clothing ID': 919.3338295291653, 'Age': 43.30047414268387}
[2024-11-04 12:45:11,136] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:MultipleImputeMissingFromInput
[2024-11-04 12:45:11,136] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2024-11-04 12:45:11,136] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (Recommended IND)
[2024-11-04 12:45:11,136] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale Recommended IND (avg=0.8202668430918514 std=0.38397562143486647 shift=0.8202668430918514 inv_scale=2.60433200488909)
[2024-11-04 12:45:11,137] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled Recommended IND (avg=8.8105360108360915e-16 std=0.9999999999999011) nulls=0
[2024-11-04 12:45:11,137] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (Positive Feedback Count)
[2024-11-04 12:45:11,137] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale Positive Feedback Count (avg=2.6301135737126473 std=5.792964725725614 shift=2.6301135737126473 inv_scale=0.17262318128041806)
[2024-11-04 12:45:11,138] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled Positive Feedback Count (avg=-3.7634222321123124e-16 std=1.0000000000000597) nulls=0
[2024-11-04 12:45:11,138] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (Clothing ID)
[2024-11-04 12:45:11,138] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale Clothing ID (avg=919.3338295291653 std=201.62590249681836 shift=919.3338295291653 inv_scale=0.004959680217752676)
[2024-11-04 12:45:11,139] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled Clothing ID (avg=-3.7809282297439586e-17 std=0.999999999999992) nulls=0
[2024-11-04 12:45:11,139] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:RescalingProcessor2 (Age)
[2024-11-04 12:45:11,139] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescale Age (avg=43.30047414268387 std=12.327406722157297 shift=43.30047414268387 inv_scale=0.08112006219463812)
[2024-11-04 12:45:11,139] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]   Rescaled Age (avg=1.5535042653537586e-17 std=1.0000000000000417) nulls=0
[2024-11-04 12:45:11,139] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FlushDFBuilder(num_flagonly)
[2024-11-04 12:45:11,139] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FlushDFBuilder(datetime_cyclical)
[2024-11-04 12:45:11,140] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FastSparseDummifyProcessor (Division Name)
[2024-11-04 12:45:11,145] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(18138, 5) nnz=18138
[2024-11-04 12:45:11,145] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FastSparseDummifyProcessor (Department Name)
[2024-11-04 12:45:11,150] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(18138, 8) nnz=18138
[2024-11-04 12:45:11,150] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FastSparseDummifyProcessor (Class Name)
[2024-11-04 12:45:11,156] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] Dummifier: Append a sparse block shape=(18138, 22) nnz=18138
[2024-11-04 12:45:11,156] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:MultipleImputeMissingFromInput
[2024-11-04 12:45:11,156] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2024-11-04 12:45:11,156] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FlushDFBuilder(cat_flagpresence)
[2024-11-04 12:45:11,156] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:MultipleImputeMissingFromInput
[2024-11-04 12:45:11,156] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2024-11-04 12:45:11,156] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:MultipleImputeMissingFromInput
[2024-11-04 12:45:11,156] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] MIMIFI: Imputing with map {}
[2024-11-04 12:45:11,156] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:FlushDFBuilder(interaction)
[2024-11-04 12:45:11,156] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:DumpPipelineState
[2024-11-04 12:45:11,156] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] ********* Pipeline state (Before feature selection)
[2024-11-04 12:45:11,156] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]    input_df= (18138, 10) 
[2024-11-04 12:45:11,156] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]    current_mf=(18138, 39) 
[2024-11-04 12:45:11,156] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]    PPR: 
[2024-11-04 12:45:11,156] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:EmitCurrentMFAsResult
[2024-11-04 12:45:11,156] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 18138
[2024-11-04 12:45:11,156] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] PROCESS WITH Step:DumpPipelineState
[2024-11-04 12:45:11,156] [14336/MainThread] [DEBUG] [dku.ml.preprocessing] ********* Pipeline state (At end)
[2024-11-04 12:45:11,156] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]    input_df= (18138, 10) 
[2024-11-04 12:45:11,156] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]    current_mf=(0, 0) 
[2024-11-04 12:45:11,156] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]    PPR: 
[2024-11-04 12:45:11,156] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]       TRAIN = <class 'dataiku.doctor.multiframe.MultiFrame'> ((18138, 39))
[2024-11-04 12:45:11,156] [14336/MainThread] [DEBUG] [dku.ml.preprocessing]       UNPROCESSED = <class 'pandas.core.frame.DataFrame'> ((18138, 10))
[2024-11-04 12:45:11,156] [14336/MainThread] [INFO] [dataiku.doctor.sparse] prepare multiframe shape=(18138,39) est_filled_columns=7 fill_ratio=0.18
[2024-11-04 12:45:11,156] [14336/MainThread] [INFO] [dataiku.doctor.sparse] Sparse matrices support is explicitly disabled, using NPA
[2024-11-04 12:45:11,367] [14336/MainThread] [INFO] [dataiku.doctor.prediction.column_importance_handler] Reusing the model feature importance
[2024-11-04 12:45:11,367] [14336/MainThread] [INFO] [dataiku.doctor.prediction.column_importance_handler] Saving column importance
[2024-11-04 12:45:11,368] [14336/MainThread] [INFO] [dataiku.doctor.prediction.background_rows_handler] Building background rows
[2024-11-04 12:45:11,384] [14336/MainThread] [INFO] [dataiku.doctor.prediction.background_rows_handler] Background estimated bias <= 0.025052088980095366
[2024-11-04 12:45:11,387] [14336/MainThread] [INFO] [dataiku.doctor.prediction.background_rows_handler] Built background rows with shape=(1000, 10)
[2024-11-04 12:45:11,387] [14336/MainThread] [INFO] [dataiku.doctor.prediction.background_rows_handler] Saving background rows with shape=(1000, 10)
[2024-11-04 12:45:11,394] [14336/MainThread] [INFO] [dataiku.doctor.utils] Coercion done
[2024-11-04 12:45:11,395] [14336/MainThread] [INFO] [dataiku.doctor.posttraining.features_distribution] Computing features distribution
[2024-11-04 12:45:11,418] [14336/MainThread] [INFO] [dataiku.doctor.posttraining.features_distribution] Saving features distribution
[2024-11-04 12:45:11,435] [14336/MainThread] [INFO] [dataiku.doctor.individual_explainer] Model supports sparse matrices, Shapley from preprocessed is not supported
[2024-11-04 12:45:11,435] [14336/MainThread] [INFO] [dataiku.doctor.individual_explainer] Using default Shapley Engine, all mentions of cells are *prior* preprocessing
[2024-11-04 12:45:11,436] [14336/MainThread] [INFO] [dataiku.doctor.prediction.explanations.shapley] 4900 cells will be generated by FrankensteinBuilder(total_num_columns=7, background_size=100, num_cols_to_explain=7) for each row
[2024-11-04 12:45:11,436] [14336/MainThread] [INFO] [dataiku.doctor.prediction.explanations.engine] Selecting batch size of: 150 (num_rows=3, max_num_batches=15, max_singly_processed_cells=300000, estimated_processed_cells=735000)
[2024-11-04 12:45:11,436] [14336/MainThread] [INFO] [dataiku.doctor.prediction.explanations.engine] Explaining 150 rows in 3 batches
[2024-11-04 12:45:11,436] [14336/MainThread] [INFO] [dataiku.doctor.prediction.explanations.engine] Explaining 50 rows (batch 1/3)
[2024-11-04 12:45:11,436] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 50
[2024-11-04 12:45:11,440] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 50
[2024-11-04 12:45:11,441] [14336/MainThread] [INFO] [dataiku.doctor.sparse] prepare multiframe shape=(50,39) est_filled_columns=7 fill_ratio=0.18
[2024-11-04 12:45:11,441] [14336/MainThread] [INFO] [dataiku.doctor.sparse] Sparse matrices support is explicitly disabled, using NPA
[2024-11-04 12:45:11,652] [14336/MainThread] [INFO] [dataiku.doctor.prediction.explanations.shapley] Built frankenstein of shape (35000, 7)
[2024-11-04 12:45:11,652] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 35000
[2024-11-04 12:45:11,691] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 35000
[2024-11-04 12:45:11,691] [14336/MainThread] [INFO] [dataiku.doctor.sparse] prepare multiframe shape=(35000,39) est_filled_columns=7 fill_ratio=0.18
[2024-11-04 12:45:11,691] [14336/MainThread] [INFO] [dataiku.doctor.sparse] Sparse matrices support is explicitly disabled, using NPA
[2024-11-04 12:45:11,971] [14336/MainThread] [INFO] [dataiku.doctor.prediction.explanations.shapley] Built frankenstein of shape (35000, 7)
[2024-11-04 12:45:11,971] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 35000
[2024-11-04 12:45:12,015] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 35000
[2024-11-04 12:45:12,015] [14336/MainThread] [INFO] [dataiku.doctor.sparse] prepare multiframe shape=(35000,39) est_filled_columns=7 fill_ratio=0.18
[2024-11-04 12:45:12,015] [14336/MainThread] [INFO] [dataiku.doctor.sparse] Sparse matrices support is explicitly disabled, using NPA
[2024-11-04 12:45:12,246] [14336/MainThread] [INFO] [dataiku.doctor.prediction.explanations.engine] Explaining 50 rows (batch 2/3)
[2024-11-04 12:45:12,246] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 50
[2024-11-04 12:45:12,250] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 50
[2024-11-04 12:45:12,250] [14336/MainThread] [INFO] [dataiku.doctor.sparse] prepare multiframe shape=(50,39) est_filled_columns=7 fill_ratio=0.18
[2024-11-04 12:45:12,250] [14336/MainThread] [INFO] [dataiku.doctor.sparse] Sparse matrices support is explicitly disabled, using NPA
[2024-11-04 12:45:12,461] [14336/MainThread] [INFO] [dataiku.doctor.prediction.explanations.shapley] Built frankenstein of shape (35000, 7)
[2024-11-04 12:45:12,461] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 35000
[2024-11-04 12:45:12,500] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 35000
[2024-11-04 12:45:12,500] [14336/MainThread] [INFO] [dataiku.doctor.sparse] prepare multiframe shape=(35000,39) est_filled_columns=7 fill_ratio=0.18
[2024-11-04 12:45:12,500] [14336/MainThread] [INFO] [dataiku.doctor.sparse] Sparse matrices support is explicitly disabled, using NPA
[2024-11-04 12:45:12,745] [14336/MainThread] [INFO] [dataiku.doctor.prediction.explanations.shapley] Built frankenstein of shape (35000, 7)
[2024-11-04 12:45:12,745] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 35000
[2024-11-04 12:45:12,785] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 35000
[2024-11-04 12:45:12,785] [14336/MainThread] [INFO] [dataiku.doctor.sparse] prepare multiframe shape=(35000,39) est_filled_columns=7 fill_ratio=0.18
[2024-11-04 12:45:12,785] [14336/MainThread] [INFO] [dataiku.doctor.sparse] Sparse matrices support is explicitly disabled, using NPA
[2024-11-04 12:45:13,013] [14336/MainThread] [INFO] [dataiku.doctor.prediction.explanations.engine] Explaining 50 rows (batch 3/3)
[2024-11-04 12:45:13,013] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 50
[2024-11-04 12:45:13,017] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 50
[2024-11-04 12:45:13,017] [14336/MainThread] [INFO] [dataiku.doctor.sparse] prepare multiframe shape=(50,39) est_filled_columns=7 fill_ratio=0.18
[2024-11-04 12:45:13,017] [14336/MainThread] [INFO] [dataiku.doctor.sparse] Sparse matrices support is explicitly disabled, using NPA
[2024-11-04 12:45:13,228] [14336/MainThread] [INFO] [dataiku.doctor.prediction.explanations.shapley] Built frankenstein of shape (35000, 7)
[2024-11-04 12:45:13,228] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 35000
[2024-11-04 12:45:13,264] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 35000
[2024-11-04 12:45:13,264] [14336/MainThread] [INFO] [dataiku.doctor.sparse] prepare multiframe shape=(35000,39) est_filled_columns=7 fill_ratio=0.18
[2024-11-04 12:45:13,264] [14336/MainThread] [INFO] [dataiku.doctor.sparse] Sparse matrices support is explicitly disabled, using NPA
[2024-11-04 12:45:13,512] [14336/MainThread] [INFO] [dataiku.doctor.prediction.explanations.shapley] Built frankenstein of shape (35000, 7)
[2024-11-04 12:45:13,512] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 35000
[2024-11-04 12:45:13,553] [14336/MainThread] [INFO] [dataiku.doctor.multiframe] Set MF index len 35000
[2024-11-04 12:45:13,553] [14336/MainThread] [INFO] [dataiku.doctor.sparse] prepare multiframe shape=(35000,39) est_filled_columns=7 fill_ratio=0.18
[2024-11-04 12:45:13,553] [14336/MainThread] [INFO] [dataiku.doctor.sparse] Sparse matrices support is explicitly disabled, using NPA
[2024-11-04 12:45:13,780] [14336/MainThread] [INFO] [dataiku.doctor.prediction.explanations.engine] Done computing explanations
[2024-11-04 12:45:13,901] [14336/MainThread] [INFO] [dataiku.doctor.utils.listener] END -  Preparing explainability
[2024-11-04 12:45:13,902] [14336/MainThread] [INFO] [dataiku.doctor.utils] Training with python version: 3.6.8 (sys.version_info(major=3, minor=6, micro=8, releaselevel='final', serial=0))
[2024-11-04 12:45:14,130] [14336/MainThread] [INFO] [dataiku.doctor.utils] Training with main package versions: {"pandas": "1.1.5", "scikit-learn": "0.20.4", "scipy": "1.2.3", "lightgbm": "3.2.1", "flask": "1.0.4", "itsdangerous": "2.0.1", "jinja2": "2.11.3", "xgboost": "1.5.2", "werkzeug": "2.0.3", "statsmodels": "0.12.2", "markupsafe": "2.0.1", "cloudpickle": "1.5.0"} 
[2024-11-04 12:45:14,130] [14336/MainThread] [INFO] [dataiku.base.socket_block_link] Client closed
[2024/11/04-12:45:14.130] [MRT-109090] [INFO] [dku.block.link.interaction]  - Check result for nullity exceptionIfNull=true result=not null
[2024/11/04-12:45:14.130] [MRT-109090] [INFO] [dku.analysis.prediction]  - Training returned ok
[2024/11/04-12:45:14.132] [MRT-109090] [INFO] [dku.ml.versioning]  - Dumping version info {"trainedWithDSSVersion":"13.2.0","trainedWithDSSConfVersion":"13200","trainedWithDSSBackend":"PY_MEMORY","backendCompatibilityVersion":18} in /data/home/samplemaster/dss_home_13/analysis-data/DKU_TUT_CUSTOMMODELING/lTMJmloT/YgfwjSKZ/sessions/s1/pp1/m1
[2024/11/04-12:45:14.134] [MRT-109090] [INFO] [dku.analysis.ml]  - preTrain={
  "algorithm": "RANDOM_FOREST_CLASSIFICATION",
  "rf_classifier_grid": {
    "n_estimators": {
      "gridMode": "EXPLICIT",
      "randomMode": "RANGE",
      "limit": {
        "min": 1
      },
      "values": [
        100
      ],
      "range": {
        "min": 80,
        "max": 200,
        "nbValues": 3,
        "scaling": "LINEAR"
      }
    },
    "n_jobs": 4,
    "max_tree_depth": {
      "gridMode": "EXPLICIT",
      "randomMode": "RANGE",
      "limit": {
        "min": 1
      },
      "values": [
        6,
        13
      ],
      "range": {
        "min": 6,
        "max": 20,
        "nbValues": 3,
        "scaling": "LINEAR"
      }
    },
    "min_samples_leaf": {
      "gridMode": "EXPLICIT",
      "randomMode": "RANGE",
      "limit": {
        "min": 1
      },
      "values": [
        1
      ],
      "range": {
        "min": 1,
        "max": 20,
        "nbValues": 3,
        "scaling": "LINEAR"
      }
    },
    "selection_mode": "sqrt",
    "max_features": {
      "gridMode": "EXPLICIT",
      "randomMode": "RANGE",
      "limit": {
        "min": 1
      },
      "values": [
        5
      ],
      "range": {
        "min": 1,
        "max": 20,
        "nbValues": 3,
        "scaling": "LINEAR"
      }
    },
    "max_feature_prop": {
      "gridMode": "EXPLICIT",
      "randomMode": "RANGE",
      "limit": {
        "min": 1.0E-23,
        "max": 1.0
      },
      "values": [
        0.3
      ],
      "range": {
        "min": 0.1,
        "max": 0.7,
        "nbValues": 3,
        "scaling": "LINEAR"
      }
    },
    "allow_sparse_matrices": false,
    "enabled": true
  },
  "max_ensemble_nodes_serialized": 6000,
  "metrics": {
    "customMetrics": [],
    "evaluationMetric": "ROC_AUC",
    "costMatrixWeights": {
      "tpGain": 1.0,
      "tnGain": 0.0,
      "fpGain": -0.3,
      "fnGain": 0.0
    },
    "liftPoint": 0.4,
    "netUpliftPoint": 0.5,
    "causalWeighting": "NO_WEIGHTING"
  },
  "autoOptimizeThreshold": true,
  "forcedClassifierThreshold": 0.0,
  "gridLength": 2,
  "grid_search_params": {
    "mode": "KFOLD",
    "splitRatio": 0.8,
    "shuffleIterations": 1,
    "nFolds": 5,
    "cvSeed": 1337,
    "stratified": true,
    "grouped": false,
    "strategy": "GRID",
    "bayesianOptimizer": "SCIKIT_OPTIMIZE",
    "randomized": true,
    "seed": 1337,
    "nIter": 0,
    "nIterRandom": 5,
    "timeout": 0,
    "nJobs": 4,
    "distributed": false,
    "nContainers": 4,
    "foldOffset": true,
    "equalDurationFolds": false
  },
  "pluginAlgoCustomGridSearch": false,
  "computeLearningCurves": false,
  "skipExpensiveReports": false,
  "propensityModeling": {
    "enabled": true,
    "calibrateProbabilities": false,
    "calibrationDataRatio": 0.1
  }
}
[2024/11/04-12:45:14.135] [MRT-109090] [INFO] [dku.analysis.ml]  - amp={
  "resolved": {
    "algorithm": "RANDOM_FOREST_CLASSIFICATION",
    "rf": {
      "estimators": 100,
      "max_tree_depth": 6,
      "min_samples_leaf": 1,
      "selection_mode": "sqrt",
      "max_features": 0,
      "max_feature_prop": 0.0
    },
    "skipExpensiveReports": false
  },
  "other": {
    "rf_min_samples_split": 3
  }
}
[2024/11/04-12:45:14.139] [MRT-109090] [WARN] [dku.kernels]  - Killing kernel python-single-command-kernel
[2024/11/04-12:45:14.139] [MRT-109090] [INFO] [dku.utils]  - Killing process PID: 14336
[2024/11/04-12:45:14.141] [MRT-109090] [INFO] [dku.engine]  - Schedule closing for WorkerPool
[2024/11/04-12:45:14.141] [MRT-109090] [INFO] [dip.tickets]  - Destroying API ticket for analysis-ml-DKU_TUT_CUSTOMMODELING-fY6u00U on behalf of admin
[2024/11/04-12:45:14.142] [AsyncCloser-127] [INFO] [com.dataiku.dip.analysis.ml.distributed.workers]  - Closing worker pool pool-q2by1oxtzsyjvkhg
[2024/11/04-12:45:14.142] [AsyncCloser-127] [INFO] [com.dataiku.dip.analysis.ml.distributed.workers]  - Unregistered worker pool: pool-q2by1oxtzsyjvkhg
[2024/11/04-12:45:14.142] [AsyncCloser-127] [INFO] [dku.engine]  - Successfully closed WorkerPool
[2024/11/04-12:45:14.168] [KNL-python-single-command-kernel-monitor-109098] [INFO] [dku.kernels]  - Process done with code 143
[2024/11/04-12:45:14.171] [KNL-python-single-command-kernel-out-109103] [DEBUG] [process]  - StreamToLine: EOF (stream closed)
[2024/11/04-12:45:14.171] [KNL-python-single-command-kernel-err-109104] [DEBUG] [process]  - StreamToLine: EOF (stream closed)
[2024/11/04-12:45:14.173] [KNL-python-single-command-kernel-monitor-109098] [WARN] [dku.resource]  - stat file for pid 14336 does not exist. Process died?
[2024/11/04-12:45:14.173] [KNL-python-single-command-kernel-monitor-109098] [DEBUG] [dku.resourceusage]  - Reporting completion of CRU:{"context":{"type":"ANALYSIS_ML_TRAIN","authIdentifier":"admin","projectKey":"DKU_TUT_CUSTOMMODELING","analysisId":"lTMJmloT","mlTaskId":"YgfwjSKZ","sessionId":"s1"},"type":"LOCAL_PROCESS","id":"78OomSa0uoDB0F0N","startTime":1730724304120,"localProcess":{"pid":14336,"commandName":"/data/home/samplemaster/dss_home_13/bin/python","cpuUserTimeMS":10690,"cpuSystemTimeMS":1600,"cpuChildrenUserTimeMS":0,"cpuChildrenSystemTimeMS":0,"cpuTotalMS":12290,"cpuCurrent":0.814,"cpuAverageOverPast60Seconds":0.0,"vmSizeMB":6144,"vmRSSMB":340,"vmHWMMB":366,"vmRSSAnonMB":294,"vmDataMB":2283,"vmSizePeakMB":6194,"vmRSSPeakMB":340,"vmRSSTotalMBS":3265,"majorFaults":310,"childrenMajorFaults":0}}
[2024/11/04-12:45:14.177] [MRT-109090] [INFO] [dku.block.link]  - Closed socket
[2024/11/04-12:45:14.177] [MRT-109090] [INFO] [dku.block.link]  - Closed socket
[2024/11/04-12:45:14.177] [MRT-109090] [INFO] [dku.block.link]  - Closed serverSocket
[2024/11/04-12:45:14.177] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.analysis.ml.python] T-YgfwjSKZ - [ct: 10464] Processing thread joined ...
[2024/11/04-12:45:14.177] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.analysis.ml.python] T-YgfwjSKZ - [ct: 10464] Joining processing thread ...
[2024/11/04-12:45:32.893] [AsyncCloser-127] [INFO] [com.dataiku.dip.analysis.ml.distributed.workers]  - Closing worker pool pool-pfiz4fkybahoqgze
[2024/11/04-12:45:32.893] [AsyncCloser-127] [INFO] [com.dataiku.dip.analysis.ml.distributed.workers]  - Unregistered worker pool: pool-pfiz4fkybahoqgze
[2024/11/04-12:45:32.894] [AsyncCloser-127] [INFO] [dku.engine]  - Successfully closed WorkerPool
[2024/11/04-12:45:32.898] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.analysis.ml.python] T-YgfwjSKZ - [ct: 29185] Processing thread joined ...
[2024/11/04-12:45:32.898] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.analysis] T-YgfwjSKZ - [ct: 29185] Train done
[2024/11/04-12:45:32.898] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.analysis.prediction] T-YgfwjSKZ - [ct: 29185] Train done
[2024/11/04-12:45:32.906] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.analysis.trainingdetails] T-YgfwjSKZ - Publishing mltask-train-done reflected event
[2024/11/04-12:45:32.907] [FT-TrainWorkThread-04bwcSMM-109089] [INFO] [dku.futures]  - [ct: 29194] Completed future execution: TrainWorkThread
