ExecutorService with a semaphore causing performance issues

ExecutorService with a semaphore causing performance issues

I have a ExecutorService with a pool size of about 10.
On some occasions the thread will require a semaphore which has 3 permits
(connections to an external system).
Not all threads need to perform this task - hence sometimes the thread
will complete without the need of an external system connection.
Eventually the thread executor ends up with 3 threads using the external
system connection and 7 more waiting on acquiring a connection.
Meanwhile, there is 20,000 more threads waiting to complete tasks which
may or may not need the external connection.
This is a waste of resources, as the other 20,000 threads may as well be
executing at this time.
I have thought about this, but can't find an appropriate solution
(possible due to my lack of experience on the concurrency libraries).
A few key notes:
It is NOT possible to identify if the thread will need the external system
connection before execution
The external system connection needs to be performed as part of the
threads transaction. In other words, there is no foreseeable way to remove
this component of work from the thread and create a separate pool of
threads.
So, is there a way I can allow 7 threads to execute other tasks while 3
use the connection to another system and X number of threads are waiting
on that connection?