Evaluation
Precision and recall
Precision is what share of the things you flagged were right; recall is what share of the things that existed you caught.
They trade against each other, and which one you want is a product decision rather than a modelling one. High precision suits cases where a false positive is expensive; high recall suits cases where a miss is.
On an imbalanced problem, accuracy is dominated by the majority class and tells you almost nothing — a classifier that always predicts negative on a five percent positive rate scores 95%.
In practice
Which one to optimise is a business decision, not a technical one. A spam filter that is wrong in the recall direction lets junk through; wrong in the precision direction it deletes an invoice. Deciding which error is more expensive before tuning is what stops the threshold being moved by whoever complained most recently.
Common questions
Should I optimise for precision or recall?
Whichever error is more expensive for you — a business decision, not a technical one. A filter wrong on recall lets junk through; wrong on precision it deletes an invoice.
What is F1 and should I use it?
The harmonic mean of the two, useful as a single summary number and misleading when the two errors have very different costs. If one direction is much worse for you, weight it explicitly instead.