Any way to resolve `export PROMPT_COMMAND='history -a'` and `HISTCONTROL=ignoredups:erasedups`?

Question

Is there any way to resolve export PROMPT_COMMAND='history -a' and HISTCONTROL=ignoredups:erasedups?

I use the former to import currently unsaved history lines when opening another Bash shell tab, but unfortunately it seems to append duplicate lines. It appears the HISTCONTROL options only have effect when closing the shell. For example, I can list history and not see duplicates, but can cat .bash_history and see them.

So I assume shopt -s histappend by default, or some other setting, saves the deduplicated history in memory or temporary storage to .bash_history only when closed.

I tried export PROMPT_COMMAND='history -w; history -a' to see if that would write the deduplicated history before appending, but that doesn't work.

Answer 1

There's no way to use ignoredups or erasedups without history -w. (Btw, the former only deletes consecutive duplicates.) This causes all shells to share the same near realtime history on every new prompt (pressing enter or interrupt) which isn't ideal many times when working on separate things in tabs.

There is a popular question on how to resolve deduplication while using a common history. Bash history: "ignoredups" and "erasedups" setting conflict with common history across sessions

I don't want a realtime common history, but I do want history saved and having it on new shell is fine. history -a does this. Using export PROMPT_COMMAND='history -a' at least saves the history every prompt. (Btw, -w writes from the file to shell list, and -a appends the list to file.)

So the only way to remove dups without realtime history is by a command. (I was going to put it into a cronjob, but it seems unnecessary now, see edit comment at bottom.)

tac $HISTFILE | cat -n | sort -k2 -k1n | uniq -f1 | sort -nk1,1 | cut -f2- | tac > $HISTFILE

file1="/etc/cron.daily/bash_history_dedup"; echo '<command>' $file1; chmod +x $file1

Might as well backup the history file with a cron job too (user,root x daily,monthly):

file1="/etc/cron.daily/backup_bash_history"; echo -e 'cp --force ~/.bash_history ~/.bash_history_bak" \nsu user -c "cp --force ~/.bash_history ~/.bash_history_bak"' > $file1 && chmod +x $file1

file1="/etc/cron.monthly/backup_bash_history"; echo -e 'cp --force ~/.bash_history ~/.bash_history_bak_monthly \nsu user -c "cp --force ~/.bash_history ~/.bash_history_bak_monthly"' > $file1 && chmod +x $file1

After deduplicating using the command, I no longer am seeing duplicates. So these may have been there from previous saving of the history list. erasedups has no effect on those.

On a side note, I really found this helpful to skip the commands with no arguments: export HISTIGNORE="!(+(*\ *))" # ignores commands without arguments. not compatible with HISTTIMEFORMAT. should be the same as grep -v -E "^\S+\s.*" $HISTFILE

Answer 2

The problem is the "atomic" appending. A middle ground would be manual history -a in the "old", and then history -n in the new bash. (This buries the existing session commands in the new one; by appending to a named file the history can be spliced)