Preserve bash history in multiple terminal windows

Question

I consistently have more than one terminal open. Anywhere from two to ten, doing various bits and bobs. Now let's say I restart and open up another set of terminals. Some remember certain things, some forget.

I want a history that:

• Remembers everything from every terminal
• Is instantly accessible from every terminal (eg if I ls in one, switch to another already-running terminal and then press up, ls shows up)
• Doesn't forget command if there are spaces at the front of the command.

Anything I can do to make bash work more like that?

Answer 1

Add the following to your ~/.bashrc:

# Avoid duplicates
HISTCONTROL=ignoredups:erasedups
# When the shell exits, append to the history file instead of overwriting it
shopt -s histappend
After each command, append to the history file and reread it
PROMPT_COMMAND="${PROMPT_COMMAND:+$PROMPT_COMMAND$'\n'}history -a; history -c; history -r"

Answer 2

So, this is all my history-related .bashrc thing:

export HISTCONTROL=ignoredups:erasedups  # no duplicate entries
export HISTSIZE=100000                   # big big history
export HISTFILESIZE=100000               # big big history
shopt -s histappend                      # append to history, don't overwrite it

# Save and reload the history after each command finishes
export PROMPT_COMMAND="history -a; history -c; history -r; $PROMPT_COMMAND"

Tested with bash 3.2.17 on Mac OS X 10.5, bash 4.1.7 on 10.6.

Answer 3

Here is my attempt at Bash session history sharing. This will enable history sharing between bash sessions in a way that the history counter does not get mixed up and history expansion like !number will work (with some constraints).

Using Bash version 4.1.5 under Ubuntu 10.04 LTS (Lucid Lynx).

HISTSIZE=9000
HISTFILESIZE=$HISTSIZE
HISTCONTROL=ignorespace:ignoredups

_bash_history_sync() {
    builtin history -a         #1
    HISTFILESIZE=$HISTSIZE     #2
    builtin history -c         #3
    builtin history -r         #4
}

history() {                  #5
    _bash_history_sync
    builtin history "$@"
}

PROMPT_COMMAND=_bash_history_sync

Explanation:

1. Append the just entered line to the $HISTFILE (default is .bash_history). This will cause $HISTFILE to grow by one line.

2. Setting the special variable $HISTFILESIZE to some value will cause Bash to truncate $HISTFILE to be no longer than $HISTFILESIZE lines by removing the oldest entries.

3. Clear the history of the running session. This will reduce the history counter by the amount of $HISTSIZE.

4. Read the contents of $HISTFILE and insert them in to the current running session history. this will raise the history counter by the amount of lines in $HISTFILE. Note that the line count of $HISTFILE is not necessarily $HISTFILESIZE.

5. The history() function overrides the builtin history to make sure that the history is synchronised before it is displayed. This is necessary for the history expansion by number (more about this later).

More explanation:

• Step 1 ensures that the command from the current running session gets written to the global history file.

• Step 4 ensures that the commands from the other sessions gets read in to the current session history.

• Because step 4 will raise the history counter, we need to reduce the counter in some way. This is done in step 3.

• In step 3 the history counter is reduced by $HISTSIZE. In step 4 the history counter is raised by the number of lines in $HISTFILE. In step 2 we make sure that the line count of $HISTFILE is exactly $HISTSIZE (this means that $HISTFILESIZE must be the same as $HISTSIZE).

About the constraints of the history expansion:

When using history expansion by number, you should always look up the number immediately before using it. That means no bash prompt display between looking up the number and using it. That usually means no enter and no ctrl+c.

Generally, once you have more than one Bash session, there is no guarantee whatsoever that a history expansion by number will retain its value between two Bash prompt displays. Because when PROMPT_COMMAND is executed the history from all other Bash sessions are integrated in the history of the current session. If any other bash session has a new command then the history numbers of the current session will be different.

I find this constraint reasonable. I have to look the number up every time anyway because I can't remember arbitrary history numbers.

Usually I use the history expansion by number like this

$ history | grep something #note number
$ !number

I recommend using the following Bash options.

## reedit a history substitution line if it failed
shopt -s histreedit
## edit a recalled history line before executing
shopt -s histverify

Strange bugs:

Running the history command piped to anything will result that command to be listed in the history twice. For example:

$ history | head
$ history | tail
$ history | grep foo
$ history | true
$ history | false

All will be listed in the history twice. I have no idea why.

Ideas for improvements:

• Modify the function _bash_history_sync() so it does not execute every time. For example it should not execute after a CTRL+C on the prompt. I often use CTRL+C to discard a long command line when I decide that I do not want to execute that line. Sometimes I have to use CTRL+C to stop a Bash completion script.

• Commands from the current session should always be the most recent in the history of the current session. This will also have the side effect that a given history number keeps its value for history entries from this session.

Answer 4

I'm not aware of any way using bash. But it's one of the most popular features of zsh.
Personally I prefer zsh over bash so I recommend trying it.

Here's the part of my .zshrc that deals with history:

SAVEHIST=10000 # Number of entries
HISTSIZE=10000
HISTFILE=~/.zsh/history # File
setopt APPEND_HISTORY # Don't erase history
setopt EXTENDED_HISTORY # Add additional data to history like timestamp
setopt INC_APPEND_HISTORY # Add immediately
setopt HIST_FIND_NO_DUPS # Don't show duplicates in search
setopt HIST_IGNORE_SPACE # Don't preserve spaces. You may want to turn it off
setopt NO_HIST_BEEP # Don't beep
setopt SHARE_HISTORY # Share history between session/terminals

Answer 5

To do this, you'll need to add two lines to your ~/.bashrc:

shopt -s histappend
PROMPT_COMMAND="history -a;history -c;history -r;$PROMPT_COMMAND"

From man bash:

If the histappend shell option is enabled (see the description of shopt under SHELL BUILTIN COMMANDS below), the lines are appended to the history file, otherwise the history file is over-written.

Answer 6

Here's an alternative that I use. It's cumbersome but it addresses the issue that @axel_c mentioned where sometimes you may want to have a separate history instance in each terminal (one for make, one for monitoring, one for vim, etc).

I keep a separate appended history file that I constantly update. I have the following mapped to a hotkey:

history | grep -v history >> ~/master_history.txt

This appends all history from the current terminal to a file called master_history.txt in your home dir.

I also have a separate hotkey to search through the master history file:

cat /home/toby/master_history.txt | grep -i

I use cat | grep because it leaves the cursor at the end to enter my regex. A less ugly way to do this would be to add a couple of scripts to your path to accomplish these tasks, but hotkeys work for my purposes. I also periodically will pull history down from other hosts I've worked on and append that history to my master_history.txt file.

It's always nice to be able to quickly search and find that tricky regex you used or that weird perl one-liner you came up with 7 months ago.

Answer 7

You can edit your BASH prompt to run the "history -a" and "history -r" that Muerr suggested:

savePS1=$PS1

(in case you mess something up, which is almost guaranteed)

PS1=$savePS1`history -a;history -r`

(note that these are back-ticks; they'll run history -a and history -r on every prompt. Since they don't output any text, your prompt will be unchanged.

Once you've got your PS1 variable set up the way you want, set it permanently it in your ~/.bashrc file.

If you want to go back to your original prompt while testing, do:

PS1=$savePS1

I've done basic testing on this to ensure that it sort of works, but can't speak to any side-effects from running history -a;history -r on every prompt.

Answer 8

Right, So finally this annoyed me to find a decent solution:

# Write history after each command
_bash_history_append() {
    builtin history -a
}
PROMPT_COMMAND="_bash_history_append; $PROMPT_COMMAND"

What this does is sort of amalgamation of what was said in this thread, except that I don't understand why would you reload the global history after every command. I very rarely care about what happens in other terminals, but I always run series of commands, say in one terminal:

make
ls -lh target/*.foo
scp target/artifact.foo vm:~/

(Simplified example)

And in another:

pv ~/test.data | nc vm:5000 >> output
less output
mv output output.backup1

No way I'd want the command to be shared

Answer 9

You can use history -a to append the current session's history to the histfile, then use history -r on the other terminals to read the histfile. 

Answer 10

If you need a bash or zsh history synchronizing solution which also solves the problem below, then see it at http://ptspts.blogspot.com/2011/03/how-to-automatically-synchronize-shell.html

The problem is the following: I have two shell windows A and B. In shell window A, I run sleep 9999, and (without waiting for the sleep to finish) in shell window B, I want to be able to see sleep 9999 in the bash history.

The reason why most other solutions here won't solve this problem is that they are writing their history changes to the the history file using PROMPT_COMMAND or PS1, both of which are executing too late, only after the sleep 9999 command has finished.

Answer 11

Here is my enhancement to @lesmana's answer. The main difference is that concurrent windows don't share history. This means you can keep working in your windows, without having context from other windows getting loaded into your current windows.

If you explicitly type 'history', OR if you open a new window then you get the history from all previous windows.

Also, I use this strategy to archive every command ever typed on my machine.

# Consistent and forever bash history
HISTSIZE=100000
HISTFILESIZE=$HISTSIZE
HISTCONTROL=ignorespace:ignoredups

_bash_history_sync() {
  builtin history -a         #1
  HISTFILESIZE=$HISTSIZE     #2
}

_bash_history_sync_and_reload() {
  builtin history -a         #1
  HISTFILESIZE=$HISTSIZE     #2
  builtin history -c         #3
  builtin history -r         #4
}

history() {                  #5
  _bash_history_sync_and_reload
  builtin history "$@"
}

export HISTTIMEFORMAT="%y/%m/%d %H:%M:%S   "
PROMPT_COMMAND='history 1 >> ${HOME}/.bash_eternal_history'
PROMPT_COMMAND=_bash_history_sync;$PROMPT_COMMAND

Answer 12

I chose to put history in a file-per-tty, as multiple people can be working on the same server - separating each session's commands makes it easier to audit.

# Convert /dev/nnn/X or /dev/nnnX to "nnnX"
HISTSUFFIX=`tty | sed 's/\///g;s/^dev//g'`
# History file is now .bash_history_pts0
HISTFILE=".bash_history_$HISTSUFFIX"
HISTTIMEFORMAT="%y-%m-%d %H:%M:%S "
HISTCONTROL=ignoredups:ignorespace
shopt -s histappend
HISTSIZE=1000
HISTFILESIZE=5000

History now looks like:

user@host:~# test 123
user@host:~# test 5451
user@host:~# history
1  15-08-11 10:09:58 test 123
2  15-08-11 10:10:00 test 5451
3  15-08-11 10:10:02 history

With the files looking like:

user@host:~# ls -la .bash*
-rw------- 1 root root  4275 Aug 11 09:42 .bash_history_pts0
-rw------- 1 root root    75 Aug 11 09:49 .bash_history_pts1
-rw-r--r-- 1 root root  3120 Aug 11 10:09 .bashrc

Answer 13

I can offer a fix for that last one: make sure the env variable HISTCONTROL does not specify "ignorespace" (or "ignoreboth").

But I feel your pain with multiple concurrent sessions. It simply isn't handled well in bash.

Answer 14

Here I will point out one problem with

export PROMPT_COMMAND="${PROMPT_COMMAND:+$PROMPT_COMMAND$'\n'}history -a; history -c; history -r"

and

PROMPT_COMMAND="$PROMPT_COMMAND;history -a; history -n"

If you run source ~/.bashrc, the $PROMPT_COMMAND will be like

"history -a; history -c; history -r history -a; history -c; history -r"

and

"history -a; history -n history -a; history -n"

This repetition occurs each time you run 'source ~/.bashrc'. You can check PROMPT_COMMAND after each time you run 'source ~/.bashrc' by running 'echo $PROMPT_COMMAND'.

You could see some commands are apparently broken: "history -n history -a". But the good news is that it still works, because other parts still form a valid command sequence (Just involving some extra cost due to executing some commands repetitively. And not so clean.)

Personally I use the following simple version:

shopt -s histappend
_bash_history_append() {
    builtin history -a
    builtin history -c
    builtin history -r
}
PROMPT_COMMAND="_bash_history_append;${PROMPT_COMMAND#"_bash_history_append;"}

which has most of the functionalities while no such issue as mentioned above.

It is good to prepend _bash_history_append to default PROMPT_COMMAND, because in default we can have something like __vte_prompt_command, which will keep current directory when opening new tabs in gnome-terminal etc.

So we remove prefix _bash_history_append from PROMPT_COMMAND and so there are no repetitions.

Another point to make is: there is really nothing magic. PROMPT_COMMAND is just a plain bash environment variable. The commands in it get executed before you get bash prompt (the $ sign). For example, your PROMPT_COMMAND is "echo 123", and you run "ls" in your terminal. The effect is like running "ls; echo 123".

$ PROMPT_COMMAND="echo 123"

output (Just like running 'PROMPT_COMMAND="echo 123"; $PROMPT_COMMAND'):

123

Run the following:

$ echo 3

output:

3
123

"history -a" is used to write the history commands in memory to ~/.bash_history

"history -c" is used to clear the history commands in memory

"history -r" is used to read history commands from ~/.bash_history to memory

See history command explanation here: http://ss64.com/bash/history.html

PS: As other users have pointed out, export is unnecessary. See: Using export in .bashrc

Answer 15

I have written a script for setting a history file per session or task its based off the following.

        # write existing history to the old file
        history -a

        # set new historyfile
        export HISTFILE="$1"
        export HISET=$1

        # touch the new file to make sure it exists
        touch $HISTFILE
        # load new history file
        history -r $HISTFILE

It doesn't necessary save every history command but it saves the ones that i care about and its easier to retrieve them then going through every command. My version also lists all history files and provides the ability to search through them all.

Full source: https://github.com/simotek/scripts-config/blob/master/hiset.sh

Answer 16

This works for ZSH

##############################################################################
# History Configuration for ZSH
##############################################################################
HISTSIZE=10000               #How many lines of history to keep in memory
HISTFILE=~/.zsh_history     #Where to save history to disk
SAVEHIST=10000               #Number of history entries to save to disk
#HISTDUP=erase               #Erase duplicates in the history file
setopt    appendhistory     #Append history to the history file (no overwriting)
setopt    sharehistory      #Share history across terminals
setopt    incappendhistory  #Immediately append to the history file, not just when a term is killed

Answer 17

Here's a solution that doesn't mix up histories from individual sessions!

Basically one has to store history of each session separately and recreate it on every prompt. Yes, it uses more resources, but it's not as slow as it may sound - delay starts to be noticeable only if you have more than 100000 history entries.

Here's the core logic:

# on every prompt, save new history to dedicated file and recreate full history
# by reading all files, always keeping history from current session on top.
update_history () {
  history -a ${HISTFILE}.$$
  history -c
  history -r
  for f in `ls ${HISTFILE}.[0-9]* | grep -v "${HISTFILE}.$$\$"`; do
    history -r $f
  done
  history -r "${HISTFILE}.$$"
}
export PROMPT_COMMAND='update_history'

# merge session history into main history file on bash exit
merge_session_history () {
  cat ${HISTFILE}.$$ >> $HISTFILE
  rm ${HISTFILE}.$$
}
trap merge_session_history EXIT

See this gist for a full solution, including some safeguards and performance optimizations.

Answer 18

I decided to start a sqlite db, for me I'm just tired of having to recreate long commands. This is just an outline but I think it could easily be made to meet your exact ask:

#!/usr/bin/env bash
# drop in bashrc
# bash history in sqlite, i hate losing commands
# run bh to see recent history
#
function create_db() (
    sqlite3 ~/.bash_db  ";" || return
sqlite3 ~/.bash_db "CREATE TABLE commands( \
    id INTEGER PRIMARY KEY AUTOINCREMENT, \
    last_used INTEGER, \
    command TEXT NOT NULL UNIQUE );" 2>/dev/null || true

sqlite3 ~/.bash_db "CREATE UNIQUE INDEX index_command ON commands(command);" 2>/dev/null || true
sqlite3 ~/.bash_db "CREATE UNIQUE INDEX index_last_used ON commands(last_used);" 2>/dev/null || true

)
function update_history() {
    LAST=builtin history 1|awk -F' ' '{for (i=2; i<=NF; i++) printf "%s ",$i}'
    DATE=$(date +%s)
    sqlite3 ~/.bash_db "UPDATE commands SET last_used='$DATE' WHERE command='$LAST';"
    sqlite3 ~/.bash_db "INSERT OR IGNORE INTO commands (last_used, command) VALUES ('$DATE', '$LAST' );"
}
function bh(){
    sqlite3 ~/.bash_db "SELECT * FROM commands ORDER BY last_used DESC LIMIT 50;"
}
create_db
PROMPT_COMMAND="update_history; $PROMPT_COMMAND"

Answer 19

Why not simply keep individual history files for each tab by using the tty ID? I don't speak bash, but with tcsh you can do this:

set histfile=~/.history_`tty|sed "s#^.*/##"`

This approach should also work in bash, probably just by:

export HISTFILE=~/.history_`tty|sed "s#^.*/##"`

As HISTFILE is evaluated at the start of each session, it will also automatically restore the associated history file on initialization.

Add to this the other features using PROMPT_COMMAND etc.

Answer 20

I long wanted this, especially the ability to retrieve a command by where it was run to re-execute in a new project (or find a directory by a command). So I put this tool together, which combines previous solutions for storing a global CLI history with an interactive grepping tool called percol (mapped to C^R). It's still slick on the first machine I started using it, now with a >2 year old CLI history.

It doesn't mess with the local CLI history as far as arrow keys are concerned, but lets you access the global history quite easily (which you can also map to something other than C^R)

Answer 21

Because I prefer infinite history which saved in custom file. I create this configuration based on https://stackoverflow.com/a/19533853/4632019:

export HISTFILESIZE=
export HISTSIZE=
export HISTTIMEFORMAT="[%F %T] "

export HISTFILE=~/.bash_myhistory
PROMPT_COMMAND="history -a; history -r; $PROMPT_COMMAND"

Answer 22

While I like being able to share history between terminals, especially new terminals. I would not want to share each and every command as it happens, as one window is often doing a specific task, separate form other windows. I would have them merge on shell exit, or when I request.

For a long time I looked for a way to merge bash history (with timestamps), and nothing seemed acceptable to me...

Finally I just 'bit the bullet' and DIY'ed a script to merge, the on-disk ".bash_history" with the in-memory shell 'history'. Preserving timestamp ordering, and command order within those timestamps.

Now when I source this (you could make it a alias or a function as you like), I use the alias 'hc'. My current shell session is merged between disk and memory, so history is updated from other previous merges, WHEN I WANT (or on logout from that shell via ".bash_logout").

Optionally you can remove unique commands (even if multi-line), and/or removing (cleaning out) simple and/or sensitive commands, according to defined perl RE's. Adjust to suit!

This is the result... https://antofthy.gitlab.io/software/history_merge.bash.txt

Enjoy.

Answer 23

This is not exactly an answer to the question, or it is. It depends on how you look at it. Apparently, bash doesn't support this out of the box. And it's not like for no reason, if you think about it. To implement this it has to write to the history file (while keeping it unduplicated) before executing every command, and read every time it needs history. Do you think a file would still suffice? Well, you might use locks, but considering that some indexing would be in order, it must be easier to just make use of some sort of database. And to top it all, do you really want all shells to have common history? Do you not use Up to reexecute command from the current shell? Do you not use sudo !!?

So what I suggest is KISS (be practical):

shopt -s histappend
HISTCONTROL=erasedups

When you want to execute a command in another shell, do history -a, then in another shell history -n, and you're good to go.

Answer 24

Here is the snippet from my .bashrc and short explanations wherever needed:

# The following line ensures that history logs screen commands as well
shopt -s histappend

# This line makes the history file to be rewritten and reread at each bash prompt
PROMPT_COMMAND="$PROMPT_COMMAND;history -a; history -n"
# Have lots of history
HISTSIZE=100000         # remember the last 100000 commands
HISTFILESIZE=100000     # start truncating commands after 100000 lines
HISTCONTROL=ignoreboth  # ignoreboth is shorthand for ignorespace and     ignoredups

The HISTFILESIZE and HISTSIZE are personal preferences and you can change them as per your tastes.