You may need to add "messageType" processor property with a value "Bytes" if you want to send a BytesMessage via JCA outbound. If you're seeing any other problem, please attach your sample application and let us know the details.
It's unclear to me why "messageType" processor property can not be specified on the inbound JCA. I'm trying to read in a BytesMessage, convert it in the bean component and send it as string via JCA outbound. Appears to be a limitation of switchyard.
You can't specify receiving message type but just receive it as it is. If the BytesMessage is in the queue, then JCA inbound receives it as a BytesMessage.