Hey Kiran,
Remove the identity source, create a sub domain with users you want to provide access to and add identity source back using a specified user in the child domain.
In authentication use username password instead of reuse session
Another approach would be to create specific groups in VC/SSO and give no permission to the domain user accounts you want to restrict permissions to
Regards
a