It sounds like YOU vs THE COMPANY here.
The first question that I need to ask is, is this a small company that might be in a temporary cash crunch?
If it were me, I would assume that there was a very good reason for "the boss" to ask ($$$$). Perhaps the company is not doing so well? Perhaps there is a feeling (in management) that the company has some nebulous claim on the miles? Remember, no one is FORCING you to work there.
Back in the early 80s Exxon put out a memo that FF miles belonged to the company. One big reason that they backed off was the nightmare associated with keeping track of personal miles vs "company" miles.