Theoretically, you can sue for libel in small claims court in most states. But there are monetary limits on small claims that would make this an unusual choice.
Small claims courts, as the name implies, generally handle small disputes with limited monetary damages. Some states do not allow lawyers to appear in small claims court but libel cases are difficult to prove and probably will require an expert. So for this and other reasons small claims court is usually unsuitable for a libel claim.