Is there a way to set the size of the history list in bash to more than 5000 lines?

Question

No matter how much I set the HISTSIZE environment variable to be larger than 5000, when printing the history list with the history builtin, it prints only the last 5000 commands. I need that because I often have a large .bash_history which exceeds 5000 lines, and sometimes one needs to address an early command by pressing Ctrl-R, but if that command is more than 5000 commands earlier, I can't access it using that mechanism. I know I can use grep on the .bash_history, but I think the Ctrl-R mechanism would be much more faster (and convenient). I use gnu bash version 4.1.

That is the full content of my .bashrc file:

    #!/bin/bash
    # ~/.bashrc: executed by bash(1) for non-login shells.
    # see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
    # for examples

    # If not running interactively, don't do anything
    [ -z "$PS1" ] && return

    # don't put duplicate lines in the history. See bash(1) for more options
    # ... or force ignoredups and ignorespace
    #HISTCONTROL=ignoredups:ignorespace:erasedups

    # append to the history file, don't overwrite it
    shopt -s histappend

    # for setting history length see HISTSIZE and HISTFILESIZE in bash(1)
    HISTSIZE=50000
    HISTFILESIZE=500000

    # check the window size after each command and, if necessary,
    # update the values of LINES and COLUMNS.
    shopt -s checkwinsize

    # make less more friendly for non-text input files, see lesspipe(1)
    [ -x /usr/bin/lesspipe ] && eval "$(SHELL=/bin/sh lesspipe)"

    # set variable identifying the chroot you work in (used in the prompt below)
    if [ -z "$debian_chroot" ] && [ -r /etc/debian_chroot ]; then
        debian_chroot=$(cat /etc/debian_chroot)
    fi

    # set a fancy prompt (non-color, unless we know we "want" color)
    case "$TERM" in
        xterm-color) color_prompt=yes;;
    esac

    # uncomment for a colored prompt, if the terminal has the capability; turned
    # off by default to not distract the user: the focus in a terminal window
    # should be on the output of commands, not on the prompt
    #force_color_prompt=yes

    if [ -n "$force_color_prompt" ]; then
        if [ -x /usr/bin/tput ] && tput setaf 1 >&/dev/null; then
        # We have color support; assume it's compliant with Ecma-48
        # (ISO/IEC-6429). (Lack of such support is extremely rare, and such
        # a case would tend to support setf rather than setaf.)
        color_prompt=yes

        else
        color_prompt=

        fi
    fi

    if [ "$color_prompt" = yes ]; then
        PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\         [\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
    else
        PS1='${debian_chroot:+($debian_chroot)}\@-\u@\h:\w\$ '
    fi
    unset color_prompt force_color_prompt

    # If this is an xterm set the title to user@host:dir
    case "$TERM" in
    xterm*|rxvt*)
        PS1="\[\e]0;${debian_chroot:+($debian_chroot)}\u@\h: \w\a\]$PS1"
        ;;
    *)
        ;;
    esac

    # enable color support of ls and also add handy aliases
    if [ -x /usr/bin/dircolors ]; then
        test -r ~/.dircolors && eval "$(dircolors -b ~/.dircolors)" || eval    "$(dircolors -b)"
        alias ls='ls --color=auto'
        #alias dir='dir --color=auto'
        #alias vdir='vdir --color=auto'

        alias grep='grep --color=auto'
        alias fgrep='fgrep --color=auto'
        alias egrep='egrep --color=auto'
    fi

    # some more ls aliases
    alias ll='ls -alF'
    alias la='ls -A'
    alias l='ls -CF'

    # Alias definitions.
    # You may want to put all your additions into a separate file like
    # ~/.bash_aliases, instead of adding them here directly.
    # See /usr/share/doc/bash-doc/examples in the bash-doc package.

    if [ -f ~/.bash_aliases ]; then
        . ~/.bash_aliases
    fi

    # enable programmable completion features (you don't need to enable
    # this, if it's already enabled in /etc/bash.bashrc and /etc/profile
    # sources /etc/bash.bashrc).
    if [ -f /etc/bash_completion ] && ! shopt -oq posix; then
        . /etc/bash_completion
    fi

Answer 1

This is the actual code that loads the history (from bashhist.c around line 260):

/* Load the history list from the history file. */
void

load_history ()
{
  char *hf;

  /* Truncate history file for interactive shells which desire it.
     Note that the history file is automatically truncated to the
     size of HISTSIZE if the user does not explicitly set the size
     differently. */
  set_if_not ("HISTSIZE", "500");
  sv_histsize ("HISTSIZE");

  set_if_not ("HISTFILESIZE", get_string_value ("HISTSIZE"));
  sv_histsize ("HISTFILESIZE");

  /* Read the history in HISTFILE into the history list. */
  hf = get_string_value ("HISTFILE");

  if (hf && *hf && file_exists (hf))
    {
      read_history (hf);
      using_history ();
      history_lines_in_file = where_history ();
    }
}

If the values of HISTSIZE and HISTFILESIZE are set, they will be used.

Readline, the library that actually handles input / line editing and history does offer facilities to put a cap on just how big the history buffer can grow. However, Bash does not place a hard ceiling on this where values any larger would be ignored, at least that I could find.

Edit

From comments, readline was indeed the culprit. I was looking (rather foolishly) at functional parameters:

there is a variable called history-size that can be read from the inputrc file. that variable sets the maximum number of history entries saved in the history list. I checked it's value in my local inputrc file to found it equal 5000. Setting it to a larger value solved the problem.

Answer 2

Your history is truncated the first time HISTSIZE is set, so if it's set to 5000 earlier in your ~/.bashrc, or in the system-wide bashrc in /etc, you need to comment those out.

Answer 3

Try both HISTFILESIZE and HISTSIZE.

Answer 4

Changing these lines in ~/.bashrc fixed it for me:

# for setting history length see HISTSIZE and HISTFILESIZE in bash(1)
HISTSIZE=5000  

HISTFILESIZE=2000

After that save the file and reload the bashrc file

$ . ~/.bashrc

Answer 5

I had the same (or a similar) problem, but inputrc was fine. In my case, the only thing that worked was commenting out HISTSIZE=1000 and HISTFILESIZE=2000 in my stock ~/.bashrc -- even though I was overriding those vars later in the same file!

Answer 6

I think you might be hitting a history ceiling on your OS for the HISTSIZE. From the man page for fc/history in Solaris 10 (running KSH):

[snip]

/usr/bin/fc

The fc utility lists or edits and reexecutes, commands pre- viously entered to an interactive sh.

The command history list references commands by number. The first number in the list is selected arbitrarily. The rela- tionship of a number to its command will not change except when the user logs in and no other process is accessing the list, at which time the system may reset the numbering to start the oldest retained command at another number (usually 1). When the number reaches the value in HISTSIZE or 32767 (whichever is greater), the shell may wrap the numbers, starting the next command with a lower number (usually 1). However, despite this optional wrapping of numbers, fc will maintain the time-ordering sequence of the commands. For example, if four commands in sequence are given the numbers 32 766, 32 767, 1 (wrapped), and 2 as they are executed, command 32 767 is considered the command previous to 1, even though its number is higher.

[snip]

which implies that the fc command can address up to 32767 entries in the history file, making it a hard ceiling for the number of commands kept in the history file. Of course YMMV, but I think you could consult your OS documentation/man pages on this matter. My 0.02...

Answer 7

My history file kept cutting off at ~4,000 lines, even after checking the system-wide and my local inputrc files. So until I figure out why it's not working, this is what I'm using:

# (try to) retain all bash history, show timestamps, and append every command
# immediately
export HISTSIZE=
export HISTFILESIZE=
export HISTTIMEFORMAT='%Y-%m-%d %T '
export HISTFILE="$HOME/.bash_history"
export PROMPT_COMMAND="history -a; $PROMPT_COMMAND"
backup whatever is new in the history file into the eternal history
function __backup_history() {
    if [ ! -f "${HISTFILE}_eternal" ]; then
        # copy the normal history file if an eternal backup doesn't exist
        cp "${HISTFILE}" "${HISTFILE}_eternal"
    else
        # update the eternal history file with any new commands
        diff "${HISTFILE}" "${HISTFILE}_eternal" 

            | tac 

            | awk '!/^</ { exit } { print }' 

            | tac 

            | sed 's/^< //' 

            | grep -v '^$' >> "${HISTFILE}_eternal"
    fi
}
__backup_history # backup on shell start
trap __backup_history EXIT # backup on shell exit

Now, every time I open a new shell or close a shell, the __backup_history() function updates the "eternal" history file with any new commands from the official history file.

In detail, what it does is:

1. Perform a diff of the two history files
2. Find only the last group of new commands that exist in ~/.bash_history but not ~/.bash_history_eternal (the last set of lines in the diff that begin with < )
3. Append those new commands to ~/.bash_eternal_history

It would probably be more ideal to put that diff command in a cron job, but I keep my ~/.bashrc synced across several computers, so this is a more portable way for me to share this setup across multiple systems. You could also add it to your $PROMPT_COMMAND if you wanted the update to take place after every command, but that seems to me like it would be adding a performance hit for no real gain.

And optimally, I would scrap this altogether and figure out why my history file is cutting off at ~4,000 lines, but alas, this works, and there are other things to worry about.