Stop nmcli from prompting for keyring password in X tray but only in the shell

Question

I want to use a little script to keep the VPN connected. Problem is that when I do nmcli c up VPN_NAME it opens an X prompt to enter the keyring password. So I add the --ask parameter but now it asks both in X and in the shell. How to get it to ask only in the shell and not in X anymore ?