Have you enabled host_alert? if not please enable it (refer to documentation) and then please check the event log - if you can't figure out, then post it here and we can then try.
Sincerely,
Here is what I would suggest:-
1) create a test.html page and embedd all the required data in it the way the payement processor requires it to be with a button to initiate the transaction
2) create a test.aspx page to catch and display the return values from payment processor
3) now test that the entire end-to-end transaction using your userid/password with the payment processor
If 3 above works fine - and displays everything as outlined by the processor then we can eliminate the possibilities (a) that there is no issue with your a/c with payment processor (b) there is no issue with your webserver, and payment processor server to communicate and pass value too-and-fro (c) sessions are properly sent-received without any loss, trimm, truncate, or drop of parameters/parameter values.
(Before doing below I would retry setting up all the values once again from scratch in catalook, reading both catalook and payment processor manual to be double sure that I have set things properly in catalook and try to do some test transactions, but if they fail then I would proceed doing the following - but thats just my way of doing things..)
Now, you will need to look into the code for this payment processor and put some host_alert event log's at: (a) just before catalook is sending the transaction to payment processor, and (b) immidiately after catalook receives the return values from the payment processor. Doing this will give the clue of what's going on and path to resolution.
Hope above helps,