Hi Demish,
Make sure the variable you are returning is java compatible value where Java should support the abap data type variable. Also, make sure the JCO connection on java end is configured properly.
Can you try to elaborate what the RFC is doing actually.
Thanks,
Srini