Myujth

Hoping I'm overlooking something stupid here or maybe I don't understand how this is working...

I have an nlp pipeline that does basically the following:

rf_pipeline = Pipeline([
('vect', TfidfVectorizer(tokenizer = spacy_tokenizer)),
('fit', RandomForestClassifier())
])

I run it:

clf = rf_pipeline.fit(X_train, y_train)
preds = clf.predict(X_test)

When I optimize I get accuracy in the high 90's with the following:

confusion_matrix(y_test, preds)
accuracy_score(y_test, preds)
precision_score(y_test, preds)

the TfidfVectorizer is the bottleneck in my computations, so I wanted to break out the pipeline. run the vectorizer, and then do a grid search on the classifier rather than running it on the whole pipeline. Here's how I broke it out:

# initialize
tfidf = TfidfVectorizer(tokenizer = spacy_tokenizer)
# transform and fit
vect = tfidf.fit_transform(X_train)
clf = rf_class.fit(vect, y_train)
# predict
clf.predict(tfidf.fit_transform(X_test))

When I took a look at the accuracy before I ran a full grid search it had plummeted to just over 50%. When I tried increasing the number of trees the score dropped almost 10%.

Any ideas?

For test set, you can't call fit_transform(), but just transform(), otherwise elements of a tfidf vectors has different meaning.

Try this

# predict
clf.predict(tfidf.transform(X_test))