Split training/validation images (Kaggle, Stanford Cars which identifies car models)
Put the images in an LMDB file, scale them to 224x224
Make some minor changes to the GoogleNet training prototxt file (drop number of classes down from 1000 to 196)
Start training. I'm not using transfer learning, although if I did, I see the same results below.
I didn't change the image mean since the calculated mean is about the same as the default. What I see is that the loss3 quickly hits ~5 and says there forever. This can't possibly be right. I'm training on an EC2 instance w/ a GPU but there's 6k images in the set.