Why does default bash seem to not have some of the commands I've used in history but does have others?

Question

On a Mac, Sierra.

I have no HIST related settings that I can see:

~ @ Magni (jcollum) 
$ env|grep HIST

~ @ Magni (jcollum) 
$ cat ~/.bash_profile | grep HIST

~ @ Magni (jcollum) 

But I have things in my history that I did yesterday that are not there after a restart. Really really doubt I put spaces before the command every time I did it.

I know there are posts on here that tell me how to fix it but I would like to know why it happens this way.

For instance when I grep for a curl that I ran yesterday, nothing:

~ @ Magni (jcollum) 
$ history | grep curl 
166  curl -X POST -d email=jollum@gmail.com -d password=********** http://localhost:3000/login
503  hg curl 
519  history | grep curl 

400 commands ago? I know I've used curl a dozen times in the past week but (and I don't use bash enough to have run hundreds of commands this week).

Basically I'm just perplexed and don't know why it's missing so much of what I did. Is it possible that history is only being recorded from one bash instance?

Answer 1

Is it possible that history is only being recorded from one bash instance?

That's kind of what's happening. The way history works in bash, unless you've gone to some lengths to change it, is:

• When bash starts, it loads saved history from the history file.
• Each time a command is executed, it's added to the history in bash's memory.
• When bash exits, it saves the accumulated history to the history file.

Now suppose you start two instances of bash. Both load the history file when they start. Both add the commands they run to their own history. When one instance exits, it saves the old history plus its own commands to the history file. When the second instance exits, it saved the old history plus its own command to the history file — and in doing this, it wipes the history from the first instance that exited.

Bash's history saving mechanism is overly simplistic. Your history lines are at the mercy of every other running instance — whichever exits last wipes the others' history. Solutions include appending the new history to the history file instead of overwriting the history file with the whole in-memory history, or appending lines to the history file as soon as they are executed. See Preserve bash history in multiple terminal windows and Preserve bash history in multiple terminal windows for some ways of implementing this and other variants.

The simplest solution is to run zsh, which saves history in a sane way by default. I recommend this, especially since you're running macOS which is stuck at an ancient version of bash because Apple doesn't like the license of versions released since 2009.