QSM has two options that can help you to limit number of attempts on your Quiz.
Limit Individual User Attempt
To limit the number of attempts on your Quiz you can set a limit of how many times a user can take the quiz.
Find more information on this option.
Limit Total Number of Entries
with this option you can limit the number of users who can take the quiz. If you set the number to 0 that means the quiz can have unlimited entries. If you set the number to 5 then only 5 users can take the quiz.