Pb with number of classed predicted

Hi,
When I started to learn less class with SSD ( 2 classes) : it diverge quickly.

Should I intiailizing my CNN differently ? beacuse my pretrained is based on 20 classes...

Or 

Is there a modification to add with my softmax function ?

Could you please give a piece of advise ?
regards