The behavior you’re describing although possible is not normal.
Usually failover on always-on clusters takes a few seconds only.
In order to troubleshoot the issue, here are some pointers where start:
When you notice this behavior, are there transactions being executed in the database?
If a transaction is open to the database, the system will try to wait until the transaction is completed. If you have long-running transactions this might lead to the behavior you’re describing. This can happen when (non-exhaustive list):
Maintenance Jobs are running in the database;
The replication cycle is taking longer than normal (usually it takes 5s or less);
Understand what’s your witness mechanism for Always-On.
Traditionally this is your Active Directory controller(s), although it’s possible to setup Always On without Active Directory. If you’re using Always-On with AD, please make sure you’ve a reliable and low-latency to your Active Directory controller. If you have multiple controllers, make sure your machine is connecting primarily the closest AD controller;