Should I go or should I stay?

Ok so a while ago I asked you guys if I should quit my job as a child carer for this family --> Read it here!

Now, I'm pretty sure that normally if you have someone looking after your kid in the afternoons and odd evenings you stick to that with them.
I haven't quit yet because I'd rather find another job first and have the excuse for better pay than '' you creep me out bye''

Today I took the kid to a place where they had trampolines and blow up castles and stuff for her to play on and the dad met us there after he'd finished whatever he needed to finish.

While she was having a hoot he asked me to sit next to him because he wanted to have a chat- which is fine, it was when he said he wants to go on holiday and would like me to go with him and his daughter to the Caribbean and that he'd pay for everything that I was confused.
I asked him about how long he was thinking of going for and he said last year he took his daughter to Disneyland in France because she really wanted to go, and the babysitter they had then came with them and they went for three weeks, and that It would be about the same amount of time this year because a week flies by when you're and holiday and he wants to do the whole tourist thing and go on excursions and stuff.

Is it normal to take your child carer on holiday with you?
I don't think so but I'd like the opinion of any parents out there because I may be over thinking everything.
I've only been working for him for a couple of months and baring in mind what I described in my previous post this is another thing to add to the creepy list.

Are there any child carers or workers out there that see this as fine?

This is the Island he's told me he wants to go to, It's called Aruba.
It does look amazing, but what if he does try something while we're there? I've got nowhere to go to, I won't know anybody, and I'll be in the same hotel as him (hopefully I'll have my own room- I didn't ask about that I just assumed I would have)

So I'd like a lot of opinions about this and ive got the whole weekend to think about it so hopefully you can help me out!