Convert a list of decimal values in a text file into hex format

Question

I have a need to convert a list of decimal values in a text file into hex format, so for example test.txt might contain:

the output should be list of hex values (hex 8 digit, with leading zeroes):

the output is printed into the text file. How to make this with python or linux shell script?

EDIT #1

I missed one extra operation: I need to add 80000000 hex to each of the created hex values. (arithmetic addition, to apply to already created list of hex values).

Possible dup - https://unix.stackexchange.com/questions/191205/bash-base-conversion-from-decimal-to-hex. — slm, Jul 21 '18 at 16:07
Do you mean add 0x80000000 to number before converting? If not, are you adding using 2's complement; how do you want overflows to be handled. i.e., what output do you want for -2147483648, -1, 0, 1 and 2147483647, (I suspect you actually want offset binary not 2's complement) — Grump, Jul 21 '18 at 22:03
To add 0x80000000 to each number after converting. 80000000 + 00020000 = 80020000 and so on. — minto, Jul 21 '18 at 22:05
... so what would -1 look like (and 2147483648 for that matter) 17FFFFFFF and 100000000? — Grump, Jul 21 '18 at 22:07
there are no negative numbers in decimal values (memory offset in decimal format). — minto, Jul 21 '18 at 22:54
how this can affect the results in this particular case? Its a decimal sequence, each subsequent number increases by 65536, up to the last 4128768. — minto, Jul 22 '18 at 12:47

jesse_b · Accepted Answer · 2018-07-21T17:25:23.997

13

You can do this using printf and bash:

printf '%08x\n' $(< test.txt)

Or using printf and bc...just...because?

printf '%08s\n' $(bc <<<"obase=16; $(< test.txt)")

In order to print the output to a text file just use the shell redirect > like:

printf '%08x\n' $(< test.txt) > output.txt

edited Jul 21 '18 at 17:25

answered Jul 21 '18 at 13:39

jesse_b

37,005

1

You should use bc as primary (only?) answer. bc is arbitrary precision, printf will fail for large numbers. – Dani_l Jul 21 '18 at 17:42
@Dani_l: I don't know of any way to reliably pad the bc output with zeros the way OP wants without using printf. – jesse_b Jul 21 '18 at 17:45
1

@Dani_l, the OP's requirement to output in 8 hex digit format will fail for large numbers. I assume he meant 8 hexit when he used 8 bit. – Grump Jul 21 '18 at 22:07
@Grump the OP's requirement is satisfied automatically by the addition operation, as the ouput would always b 8 or more digits, and reducing the number of digits doesn't make sense in an addition operation. – Dani_l Jul 22 '18 at 07:36
exactly, the requirement is 8 digits, not 8 or more. – Grump Jul 22 '18 at 09:45

score 5 · Answer 2 · 2018-07-22T18:50:30.263

5

Three possible solutions (assuming each line is a set of only digits):

For shells like ksh, bash, zsh:

printf '%08x\n' $(<infile)

Only for bash

<file.txt mapfile -t arr;   printf '%08x\n' "${arr[@]}"

For simpler shells: dash (the default sh in Debian based systems), ash (busybox emulated shell), yash and some default shells in AIX and Solaris:

printf '%08x\n' $(cat infile)

In fact, for a shell like the heirloom version (Bourne like) it needs to be written like (which do work on all posix shells listed above but I strongly recommend to do not use it ):

$ printf '%08x\n' `cat infile`

Answer to EDIT #1

Understand that an hex value of 80000000 will cause overflow on a 32 bit computer (not common this days, but possible). Check that echo "$((0x80000000))" do not print a negative value.

$ for i in $(<infile); do printf '%08x\n' "$(($i+0x80000000))"; done
80020000
80030000
80040000
80050000
80060000

edited Jul 22 '18 at 18:50

answered Jul 21 '18 at 14:11

Probably should use while read instead of for loop of file contents. https://unix.stackexchange.com/questions/169716/why-is-using-a-shell-loop-to-process-text-considered-bad-practice ; https://mywiki.wooledge.org/DontReadLinesWithFor – Dani_l Jul 22 '18 at 11:54
1

As I said in the first line: "assuming each line is a set of only digits" it makes no difference to add the complexity of read. @Dani_I – Jul 22 '18 at 18:56
@Isaac maybe some clarification should be done regarding hex value of 80000000 that may cause potential overflow on a 32 bit computer. 80000000 is the base address of the flash chip in the embedded system. Base address chosen by device manufacturer, so likely allowed by the processor’s memory controller. I'm not sure it can cause overflow in this case. – minto Jul 23 '18 at 10:14
@minto In answer to your comment I'm not sure it can cause overflow in this case I have to say: I do not know if you are running the shell inside the same system that use that memory address. You may be developing and compiling on one computer and then sending the compiled code to the embedded system. Or ... you may be running the shell inside the embedded system, in which case it seems reasonable to expect that the shell has been compiled to run in that system and that the shell has been compiled with the same bit size. But I can not know that both facts are true. – Jul 23 '18 at 11:49

Kusalananda · Answer 3 · 2018-07-21T16:45:21.680

5

Using awk:

$ awk '/[0-9]/ { printf("%08x\n", $0) }' file
00020000
00030000
00040000
00050000
00060000

edited Jul 21 '18 at 16:45

answered Jul 21 '18 at 14:33

Kusalananda

333,661

1

This code add extra 8-bit at the end 00000000. – minto Jul 21 '18 at 16:17
@minto Did you run the command on a file that had ... on the last line? If so, then yes. It would convert all values to hexadecimal, even empty lines. Data that can not be converted to an integer would be taken as zero. – Kusalananda Jul 21 '18 at 16:34
yes, the file have an empty newline at end of file, but empty, no dots. – minto Jul 21 '18 at 16:44
@minto See updated answer. The updated answer outputs only if the input line contains a number. How big is your file? – Kusalananda Jul 21 '18 at 16:46
File have just 64 lines. – minto Jul 21 '18 at 16:48

score 2 · Answer 4 · answered Jul 21 '18 at 18:34

2

Yet another one-liner. Still not python, but which allows comments, empty lines and whatever in the file, and only prints out the result for lines which contain only a number.

perl -ne '/^(\d+)$/ && printf "%08x\n", $1'  $your_file

Given a file like this:

$ cat $your_file
# some numbers,
131072
196608

262144
327680
393216

and empty lines
and whatever...

It prints

answered Jul 21 '18 at 18:34

mivk

3,596

How to add 80000000 hex value to each of the created hex values? I mean something like printf "%X\n" $((80000000 + 0x$b)) b=list of hex values – minto Jul 21 '18 at 21:49
@minto : If I understand what you mean, you could replace $1 in the example above with $1+hex("80000000") or $1+2147483648. The output would then be 80020000, 80030000, etc. – mivk Jul 21 '18 at 23:05
but I have no perl. – minto Jul 21 '18 at 23:13
You used linux in your tags. I know of no Linux (or Unix) distribution which doesn't come with Perl in the base install. Even Mac OS X has it. – mivk Jul 21 '18 at 23:23

Zombo · Answer 5 · 2021-04-05T21:53:48.853

1

Here are a couple of one line solutions in PHP:

<?php
foreach (file('test.txt') as $s) echo dechex(+$s), "\n";

<?php
array_map(fn($s) => print dechex(+$s) . "\n", file('test.txt'));

edited Apr 05 '21 at 21:53

answered Jul 22 '18 at 04:44

Zombo

1
5
44
63

score 1 · Answer 6 · edited Jul 22 '18 at 18:33

Using julia

$ julia -e 'function hexadd(x) hex(( x + 0x80000000),8) end ; output = open("output.txt","w") ; open("test.txt") do inputfile for num in eachline(inputfile) write(output,"$(hexadd(parse(Int,num)))\n") end end'

It should be easier to read multiline:

function hexadd(x)
    hex(( x + 0x80000000),8)
end
output = open("output.txt","w")
open("test.txt") do inputfile
    for num in eachline(inputfile)
        write(output,"$(hexadd(parse(Int,num)))\n")
    end
end

hex( value, pad) converts a value to hex, and adds padding as needed.
hexadd requires integer as input so we convert strings (output of eachline) to Int with parse(Int,"string") and use that in our function.
julia -e evaluates an expression.
julia can be installed natively in fedora,ubuntu

Using dc:

$ dc -f test.txt -e '16o16i[80000000+psaz1<r]dsrx80000000+p' > output.txt

Explanation:

dc -f test.txt -e '16o 16i [80000000 + p sa z 1 <r] sr z 1 <r 80000000 + p' > output.txt
-f file push contents of file to the stack
-e execute expression
16o convert output to base 16
16i assume input in base 16. This happens after reading the file, so the file was read in base 10. More explicitly I could do dc -e '10i' -f file -e '16 i 8000000'
[...] push a string to the stack. This string will be used as a macro.
80000000 + add the current top of stack and hex 80000000. Push the result to top of stack.
p Print current top of stack, without popping. Only print option that prints a newline.
sa Pop and store top of stack in register 'a'. It never gets used, just a way to get rid of the top of sack.
z The current top of stack now has the stack depth. Required to end the recursive call.
1 push 1 to top of stack. Useful in comparison to stack depth (z) pushed earlier.
<r compare 2 values from stack, if 2nd is less than 1st, execute register 'r'. In effect compare stack depth and '1'.
sr pop and store in register 'r'. Now the macro is in register 'r', and will execute as long as stack depth greater than 1. Except nothing has called it yet.
d Duplicate top of stack and push.
x Execute top of stack as macro.
d sr x Duplicate macro and push, now stack has 2 copies, pop and store top copy to register, pop and execute 2nd copy...
So push stack depth and 1, compare and execute, and for last element, again add the hex value and print.