How to write contents of a file to new file removing repeated lines

Question

For example, a file file1.txt contains

Hi how are you  
hello  
today is monday  
hello  
I am fine  
Hi how are you 

After the processing of file1.txt it should write to file2.txt and contents should be like this without repeating the same lines.

Hi how are you  
hello  
today is monday  
I am fine  

What command can I use to do that in linux terminal?

Answer 1

start cmd:> awk 'lines[$0]++ == 0' input
Hi how are you
hello  
today is monday  
I am fine

Answer 2

This is an easy job for sort, use the unique (-u) option of sort:

% sort -u file1.txt
hello
Hi how are you
I am fine
today is monday

To save it in file2.txt:

sort -u file1.txt >file2.txt

---

If you want to preserve the initial order:

% nl file1.txt | sort -uk2,2 | sort -k1,1n | cut -f2
Hi how are you
hello
today is monday
I am fine