Why does my .bash_history contain duplicate segments?

Question

Since I keep my bash history under source control, I've noticed that sometimes sizable segments of the history end up repeated, sometimes hours or days after their original execution. I use Debian 7.7 and have the following config:

shopt -s histappend

export HISTCONTROL=ignoreboth:erasedups
export HISTSIZE=1000000
export HISTFILESIZE=1000000

I suspect there is some interaction between multiple terminals, histappend, and erasedups. I'm answering this question myself but if someone disagrees or has more detail I would like other answers!

Edit: I believe this is not a duplicate -- there are many questions asking how to ignore duplicate entries; I'm asking about getting rid of a buggy behavior around mistakenly duplicated history segments. (Whole chunks repeated that I had actually executed only once.)

Answer 1

I have experimented with removing erasedups and that seems to do the trick. My guess is that when duplicates are erased, the local copy of history gets out of sync with the count of items to be appended to the history file and therefore too much is pulled from history during appending.

Answer 2

My best bet is that since you set ignoreboth new duplicates are just ignored; the old duplicates are kept in the history. Try using

HISTCONTROL=ignorespace:erasedups