Ok, I am guessing you have an electric guitar... this should work with a accoustic as well it is just how you mic the guitar up that might be different.
You would plug the mic into the XLR/jack input on the recorder. You would then place the mic appropriately to record from your amplifier. (place the mic just off centre and direct it slightly towards the centre, if that makes sense...
) You would then hit the record button on the recorder and Bob is your uncle...
For recordng a bass guitar you would rather use the line out from the bass amplifier to record. plug the line out from the bass amplifier into the jack input of hte recorder. Hit the record butotn on the channel you wish to record and you're waxed...
You must remember to enalbe only the track that you want to record at any point in time. If you dont enable only the track you wish to record on you might actually record straight over that cool Kirk Hammet solo you just did perfectly in 105 takes...
You dont really need to have a mixer in your setup but you can actually use it. All you have to do is plug the instruments into the mixer, then you need to go form either a auxilary channel or an insert to the recorder. I cant really see the benefit in this, except if you are using the mixer as away to monitor what you are playing...
I hope this helps... I dont know if this explanation would assit you or confuse you more...