Applying 4o Vision Finetuning to Chemistry Diagrams

The task is to extract student’s attempts for chemistry diagram questions.

These diagrams are a graph with nodes and edges.

Using VLLMs out of the box often results in the model correcting the chemistry equations or missing key notation.

Here we explore Vision Finetuning, and see how far we can go with less than 10 hand-labelled examples.

Kuang Wen and I will show the data we have, the augmentation techniques, and our current demo app comparing finetuned with non-finetuned.