• 0 Posts
  • 3 Comments
Joined 1 year ago
cake
Cake day: October 27th, 2023

help-circle


  • If it’s just about classifying documents nothing stops you from iteratively hand labeling a new set of documents, training a model to suggest classes, correcting the model predictions on a new set of documents and repeat. Will be cheaper than using complex models and may get you there. If it’s about making the whole stuff indexable best bet is to use a vector database, sentence transformer embeddings and then putting the top 4-5 closest paragraphs to your search into a LLM prompt for reranking. If it’s about extracting structure information from each document you will most likely need to use a) specialized custom model or b) one LLM prompt for each document