Comparative Fine-Tuning of GPT-2 on Question Answering and Dialogue Datasets for Medical Text Generation

Fine-tuning large language models (LLM) using medical data-sets presents significant opportunities for developing reliable and informative AI-driven health applications. This research investigates how different dataset structures (formatted question-answer (QA) pairs versus conversational doctor-patient dialogues) influence the effectiveness of a GPT-2-based generative model. Models trained on each dataset were evaluated using established NLP metrics (BLEU, ROUGE-1, ROUGE-L, BERTScore) and qualitative evaluations covering sentiment alignment, factual consistency (assessed via natural language inference), and readability. The results indicate that the QA-trained model achieves superior performance in semantic accuracy and sentiment alignment compared to the dialogue-based model, which produced responses that were marginally more readable. However, both models exhibited notably low factual entailment scores, highlighting an essential area for further improvement. These insights emphasize the importance of cautious dataset selection and model assessment strategies in clinical NLP. They also suggest promising directions for enhancing factual accuracy, domain specificity, and explanatory capabilities in future research.