[Question]: Why output arrow instead of string like "left, right"

[TianzeXie]

Question

I think the string output could leverage the base model's ability efficiently because the base model is already understand the meaning of these string and these string could align with the input instruction when such string exist in it, is it?

[Tai-Wang]

It is a design inherited from StreamVLN and is just used to save output tokens for faster inference. I agree with the idea that the string output can yield better alignment with the pretrained behavior. You can feel free to have a try to replace it during training, and I am also curious about what would happen :)

[TianzeXie]

It is a design inherited from StreamVLN and is just used to save output tokens for faster inference. I agree with the idea that the string output can yield better alignment with the pretrained behavior. You can feel free to have a try to replace it during training, and I am also curious about what would happen :)

got it, thank you. I tried it and the loss converged much faster