Comments (17)
laurenyu
commented on August 27, 2024
Hi, thanks for using SageMaker!
Could you post your code and (if these exist) any relevant log messages from CloudWatch?
from sagemaker-python-sdk.
samuelhkahn
commented on August 27, 2024
Thanks for the quick reply @laurenyu !
Here is the script I used in my hosted notebook:
# Define IAM role
import boto3
import re
import os
import numpy as np
import pandas as pd
from sagemaker import get_execution_role
role = get_execution_role()
import sagemaker as sage
from time import gmtime, strftime
sess = sage.Session()
account = sess.boto_session.client('sts').get_caller_identity()['Account']
region = sess.boto_session.region_name
image = '{}.dkr.ecr.{}.amazonaws.com/PLACEHOLDER'.format(account, region)
tree = sage.estimator.Estimator(image,
role, 1, 'ml.m4.10xlarge',
output_path="s3://{}".format(sess.default_bucket()),
sagemaker_session=sess)
tree.fit("s3://PLACEHOLDER")
Here is my training script which got packaged in my Docker container. Note training worked fine on a small subset of data. Also running on my local machine, each process takes ~13.4 G of mem:
!/usr/bin/env python
# coding: utf-8
from __future__ import print_function
import pandas as pd
import numpy as np
import os
import json
import pickle
import sys
import traceback
from sklearn.base import BaseEstimator, TransformerMixin
import string
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.pipeline import Pipeline,FeatureUnion
from sklearn.linear_model import LogisticRegression
from sklearn.linear_model import SGDClassifier
from sklearn.grid_search import GridSearchCV
from sklearn.model_selection import RandomizedSearchCV
from sklearn.metrics import confusion_matrix
from sklearn.metrics import f1_score,precision_score,recall_score,classification_report
from sklearn.utils import shuffle
from sklearn.model_selection import train_test_split
from sklearn.externals import joblib
from DollarPercentSigns import DollarPercentSigns
from SourceTransformer import SourceTransformer
from SelectColumnsTransfomer import SelectColumnsTransfomer
# These are the paths to where SageMaker mounts interesting things in your container.
prefix = '/opt/ml/'
input_path = prefix + 'input/data'
output_path = os.path.join(prefix, 'output')
model_path = os.path.join(prefix, 'model')
param_path = os.path.join(prefix, 'input/config/hyperparameters.json')
# This algorithm has a single channel of input data called 'training'. Since we run in
# File mode, the input files are copied to the directory specified here.
channel_name='training'
training_path = os.path.join(input_path, channel_name)
# The function to execute the training.
def train():
print('Starting the training.')
try:
data=pd.read_pickle("/opt/ml/input/data/training/financial_data.pkl")
data2=pd.read_pickle("/opt/ml/input/data/training/positive_dataframe")
data3=pd.read_pickle("/opt/ml/input/data/training/financial_training_dataset.pkl")
data2["source"]=""
data2["financial"]=data2["target"]
data2=data2[data.columns.tolist()]
data3["source"]=""
data3["financial"]=data3["target"]
data3=data3[data.columns.tolist()]
positive_classes=data3[data3["financial"]==1]
data=data.append(positive_classes).append(data2)
financial_sources=data[data['financial']==1]['source'].tolist()
not_financial_sources=data[data['financial']==0]['source'].tolist()
np.random.seed(0)
data=shuffle(data)
# train,dev = train_test_split(data,test_size=0.1)
param_grid={'features__text_tfidf__tf_idf_ngram__ngram_range':[ (1,3)],
'features__transformer_weights':[{'title_tfidf_weight': 1,'text_tfidf_weight': 1,"source_weight":1},
{'title_tfidf_weight': 2,'text_tfidf_weight': 1,"source_weight":1},
{'title_tfidf_weight': 3,'text_tfidf_weight': 2,"source_weight":1},
{'title_tfidf_weight': 1,'text_tfidf_weight': 2,"source_weight":1},
{'title_tfidf_weight': 1,'text_tfidf_weight': 3,"source_weight":1}],
'features__title_tfidf__tf_idf_ngram__ngram_range':[(1,3)],
'classifier__penalty':["l1","l2"] ,
'classifier__C':[0.1,0.5,1] ,
"features__text_tfidf__tf_idf_ngram__max_features":[150000,250000,350000],
"features__title_tfidf__tf_idf_ngram__max_features":[150000,250000,350000],
}
pipeline = Pipeline([
('features', FeatureUnion([
('dollar_sign', Pipeline([
('selector', SelectColumnsTransfomer("title")),
('tf_idf_ngram', DollarPercentSigns())
])),
('text_tfidf', Pipeline([
('selector', SelectColumnsTransfomer("text")),
('tf_idf_ngram', TfidfVectorizer())
])),
('title_tfidf', Pipeline([
('selector', SelectColumnsTransfomer("title")),
('tf_idf_ngram', TfidfVectorizer())
])),
('source', Pipeline([
('selector', SelectColumnsTransfomer("source")),
('source_binarizer', SourceTransformer())
])),
]
)
),
('classifier', LogisticRegression(class_weight="balanced"))
])
grid=GridSearchCV(pipeline,cv=3,param_grid=param_grid,n_jobs=4,verbose=2,pre_dispatch=4,scoring="precision")
grid.fit(data,data["financial"])
best_clf=grid.best_estimator_
best_clf.fit(data,data["financial"])
joblib.dump(best_clf,"/opt/ml/model/financial_model.pkl")
print('Training complete.')
except Exception as e:
# Write out an error file. This will be returned as the failureReason in the
# DescribeTrainingJob result.
trc = traceback.format_exc()
with open(os.path.join(output_path, 'failure'), 'w') as s:
s.write('Exception during training: ' + str(e) + '\n' + trc)
print('Exception during training: ' + str(e) + '\n' + trc)
# Printing this causes the exception to be in the training job logs, as well.
# A non-zero exit code causes the training job to be marked as Failed.
sys.exit(255)
if __name__ == "__main__":
train()
# A zero exit code causes the job to be marked a Succeeded.
sys.exit(0)
The training script closely mimics one of the example scripts you guys included in "amazon-sagemaker-examples".
Here is the logs:
/usr/local/lib/python2.7/dist-packages/sklearn/cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
"This module will be removed in 0.20.", DeprecationWarning)
/usr/local/lib/python2.7/dist-packages/sklearn/grid_search.py:42: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. This module will be removed in 0.20.
DeprecationWarning)
Starting the training.
Fitting 3 folds for each of 270 candidates, totalling 810 fits
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 2, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -13.9min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 2, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -13.9min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 2, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 2, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.0min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.3min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 3, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 3, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 2, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.5min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 2, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.2min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 3, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 3, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.1min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 3, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.2min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 3, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.9min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.9min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -14.9min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.3min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 3, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 3, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 3, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 3, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -14.6min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 3, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -14.9min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 3, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -14.9min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.1min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 2, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 2, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -14.7min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -14.7min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 2, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 2, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.1min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 3, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 2, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.2min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 3, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 3, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 3, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -14.6min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 2, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.3min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 3, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -14.6min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 3, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.2min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 3, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.4min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.0min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.5min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 3, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 3, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 3, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -14.7min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 3, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.3min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=250000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 3, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.4min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.1min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 2, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 2, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.2min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 2, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 2, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.8min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 2, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.8min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 3, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 3, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 2, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.7min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 3, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 3, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.7min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 3, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.8min
[Parallel(n_jobs=4)]: Done 37 tasks | elapsed: 158.1min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 3, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.3min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.3min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 3, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -14.2min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 3, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -13.4min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 3, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 3, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 3, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -14.1min
[CV] features__text_tfidf__tf_idf_ngram__max_features=150000, features__title_tfidf__tf_idf_ngram__max_features=350000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 3, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -13.9min
[CV] features__text_tfidf__tf_idf_ngram__max_features=250000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=250000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=250000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -13.6min
[CV] features__text_tfidf__tf_idf_ngram__max_features=250000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=250000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -14.5min
[CV] features__text_tfidf__tf_idf_ngram__max_features=250000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 2, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=250000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -14.5min
[CV] features__text_tfidf__tf_idf_ngram__max_features=250000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 2, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -14.4min
[CV] features__text_tfidf__tf_idf_ngram__max_features=250000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 2, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=250000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 2, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=250000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 1, 'title_tfidf_weight': 2, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -14.3min
[CV] features__text_tfidf__tf_idf_ngram__max_features=250000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 3, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=250000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 3, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -14.3min
[CV] features__text_tfidf__tf_idf_ngram__max_features=250000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 3, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=250000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 3, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -14.6min
[CV] features__text_tfidf__tf_idf_ngram__max_features=250000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 3, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
[CV] features__text_tfidf__tf_idf_ngram__max_features=250000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 3, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1 -15.3min
[CV] features__text_tfidf__tf_idf_ngram__max_features=250000, features__title_tfidf__tf_idf_ngram__max_features=150000, features__text_tfidf__tf_idf_ngram__ngram_range=(1, 3), features__transformer_weights={'text_tfidf_weight': 2, 'title_tfidf_weight': 1, 'source_weight': 1}, features__title_tfidf__tf_idf_ngram__ngram_range=(1, 3), classifier__C=0.1, classifier__penalty=l1
There is nothing in the CloudWatch logs after the above fit. Let me know if there is anything else that you need.
from sagemaker-python-sdk.
samuelhkahn
commented on August 27, 2024
Ran it again overnight and ran into the same error. One of my fits did seem to run out of memory, but this still doesn't explain the Internal Server Error above. I added some logging to give better error descriptions, but I see nothing new in the logs with the error is thrown. This is kind of a show stopper for me right now unfortunately, since I can't tune a model for prod.
12:01:33
Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "/usr/lib/python2.7/multiprocessing/pool.py", line 326, in _handle_workers pool._maintain_pool() File "/usr/lib/python2.7/multiprocessing/pool.py", line 230
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 326, in _handle_workers
pool._maintain_pool()
File "/usr/lib/python2.7/multiprocessing/pool.py", line 230, in _maintain_pool
self._repopulate_pool()
File "/usr/lib/python2.7/multiprocessing/pool.py", line 223, in _repopulate_pool
w.start()
File "/usr/lib/python2.7/multiprocessing/process.py", line 130, in start
self._popen = Popen(self)
File "/usr/lib/python2.7/multiprocessing/forking.py", line 121, in __init__
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
from sagemaker-python-sdk.
samuelhkahn
commented on August 27, 2024
Just an update, went ahead and ran the GridSearchCV with no parallelism, i.e. setting n_jobs=1, and still getting the Internal Server Error.
from sagemaker-python-sdk.
rama0120
commented on August 27, 2024
Hi,
Thank you for using SageMaker!
Apologies about the long silence from our end. We have diagnosed the platform issue on our end and it was fixed yesterday. Please retry your job, it should continue without any issues.
Kindly note that there is a 5-day maximum for training jobs right now - https://docs.aws.amazon.com/sagemaker/latest/dg/API_StoppingCondition.html#SageMaker-Type-StoppingCondition-MaxRuntimeInSeconds
Please reach back to us either here or via AWS customer service if you have unresolved issues.
from sagemaker-python-sdk.
samuelhkahn
commented on August 27, 2024
Thanks for the response and I think we all appreciate how responsive the AWS Sagemaker team is! I just started running the job again, i'll let you guys know if things run smoothly.
from sagemaker-python-sdk.
nadiaya
commented on August 27, 2024
Hi, we hope this fix worked.
We are going to close this issue in 7 days if we don't hear back from you.
Please, let us know if the problem still persists.
from sagemaker-python-sdk.
gaceladri
commented on August 27, 2024
Same error here. When the training of the deepar demo finish, it gets me an error like this:
[03/02/2018 15:27:25 INFO 140691788121920] Final loss: -0.0579527433102 (occured at epoch 26)
#metrics {"Metrics": {"get_graph.time": {"count": 1, "max": 10938.009977340698, "sum": 10938.009977340698, "min": 10938.009977340698}}, "EndTime": 1520004456.036648, "Dimensions": {"Host": "algo-1", "Operation": "training", "Algorithm": "AWS/DeepAR"}, "StartTime": 1520004445.098425}
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-20-86f3ce5e5efb> in <module>()
4 }
5
----> 6 estimator.fit(inputs=data_channels)
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/estimator.py in fit(self, inputs, wait, logs, job_name)
152 self.latest_training_job = _TrainingJob.start_new(self, inputs)
153 if wait:
--> 154 self.latest_training_job.wait(logs=logs)
155 else:
156 raise NotImplemented('Asynchronous fit not available')
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/estimator.py in wait(self, logs)
321 def wait(self, logs=True):
322 if logs:
--> 323 self.sagemaker_session.logs_for_job(self.job_name, wait=True)
324 else:
325 self.sagemaker_session.wait_for_job(self.job_name)
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in logs_for_job(self, job_name, wait, poll)
645
646 if wait:
--> 647 self._check_job_status(job_name, description)
648 if dot:
649 print()
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in _check_job_status(self, job, desc)
388 if status != 'Completed':
389 reason = desc.get('FailureReason', '(No reason provided)')
--> 390 raise ValueError('Error training {}: {} Reason: {}'.format(job, status, reason))
391
392 def wait_for_endpoint(self, endpoint, poll=5):
ValueError: Error training DEMO-deepar-2018-03-02-15-16-09-505: Failed Reason: InternalServerError: We encountered an internal error. Please try again.
from sagemaker-python-sdk.
andremoeller
commented on August 27, 2024
Hi @gaceladri ,
Sorry for the delayed response. We're looking into why you received an InternalServerError when running the DeepAR algorithm. Thanks!
from sagemaker-python-sdk.
winstonaws
commented on August 27, 2024
InternalServerError means that an unknown failure condition occurred, so we need information on your specific job to debug. However, you should not post your account information here. Please private message it to me through the AWS forums: https://forums.aws.amazon.com/profile.jspa?userID=395850 And then post here to let me know that you've done so.
Remember to only send your information to verified Amazonians! Look for the Amazon icon next to the username in the profile.
We need the following information:
- account ID
- training job name
- sagemaker region
from sagemaker-python-sdk.
Dickmannma
commented on August 27, 2024
I am having the same issue. It looks like you looked at the same demo I did.
For Sagemaker team: check out this demo: LINK
Same error as listed above.
from sagemaker-python-sdk.
yangaws
commented on August 27, 2024
InternalServerError means that an unknown failure condition occurred, so we need information on your specific job to debug. However, you should not post your account information here. Please private message it to me through the AWS forums: https://forums.aws.amazon.com/profile.jspa?userID=435518 And then post here to let me know that you've done so.
Remember to only send your information to verified Amazonians! Look for the Amazon icon next to the username in the profile.
We need the following information:
account ID
training job name
sagemaker region
from sagemaker-python-sdk.
ezra-freedman
commented on August 27, 2024
I am also getting an internal server error, running training with a custom docker image. I posted to the sagemaker forum several weeks ago (https://forums.aws.amazon.com/thread.jspa?threadID=282729&tstart=0) but have not received a reply.
At first I thought I was running against the 5-day time limit but I get the error after running for 3 hours on 20 instances or on 62 instances. Also of note is that I can run the job to completion if I run as 62 separate jobs (but then I need to monitor all jobs and stitch together the artifacts manually upon completion).
Could you help provide insight?
Thanks.
from sagemaker-python-sdk.
jesterhazy
commented on August 27, 2024
@ezra-freedman I just replied to your forum post. I'll need your account id and other info to assist, so please check out my reply and message me there.
from sagemaker-python-sdk.
mattwinkler
commented on August 27, 2024
Hello - I'm encountering a problem similar to the one above using the AWS implementation of Random Cut Forest. Can you please message me to collect details?
from sagemaker-python-sdk.
namheegordonkim
commented on August 27, 2024
Reopening this: specifying wrong hyperparameters for an ECR image results in InternalServerError without referencing the actual issue. Logs are missing and the training job page treats it as an "unhealthy instance" problem.
from sagemaker-python-sdk.
namheegordonkim
commented on August 27, 2024
specifying wrong hyperparameters for an ECR image results in InternalServerError without referencing the actual issue.
False alarm; turns out my reference to the ECR image had a typo. The error happens within the "Preparing the Instances" stage, presumably docker pulling the image. However, some informative logs would be very helpful as the typos are not always super easy to catch.
from sagemaker-python-sdk.
Related Issues (20)
- How to recreate the predictor variable from the endpoint_name if somehow the variable is lost?
- No DataProcessing in local transform job HOT 1
- Update documentation to use `sagemaker-python-sdk` as opposed to `sagemaker` HOT 3
- Fix documentation of FeatureGroup HOT 1
- Sagemaker Local-Mode permission errors
- ModelStep name includes "-RegisterModel" suffix HOT 1
- Tags are't inherited from the `sagemaker_config` to hyperparameter tuning requests
- ValueError: Invalid explanations value when creating ModelCard from ModelPackage
- AsyncPredictor fails if name is None, despite it being the default
- Newline chars are not supported in sagemaker feature store string types
- SageMaker Batch currently doesn't support Model entity with container definitions which use ModelDataSource attribute
- 'base_job_name' parameter is not used to name the Processing Jobs that are created by the Model Monitoring Schedule
- (AutoMLV2) UnboundLocalError: local variable 'input_config' referenced before assignment
- Import/get data from bigquery
- Endless loop when try to deploy a sagemaker pipeline from a windows machine HOT 2
- HuggingFaceProcessor with ProcessingStep results in import errors (similar to issues/2656) HOT 3
- Error in using Clarify with AWS Jumpstart image classification models
- Forbidden Access to S3 bucket after domain initialization
- Can I create lambda step from ecr image by using Lambda class in lambda_help.py?
- sagemaker.model_monitor.DefaultModelMonitor suggest_baseline is not able to read Japanese text HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sagemaker-python-sdk.