Because the only thing you do is to change the visibility of your div which contains the youtube player, the video will keep playing till end. You need to use youtube javascript player api to control the player and stop video with command "player.stopVideo():Void" when user closes the modal popup.
Please review http://code.google.com/apis/youtube/js_api_reference.html#Functions for further information and examples.
