7

I've just discovered that Unix time does not include the leap seconds. I think it's quite surprising because doing that, it is slowly diverging from UTC... but that's not the point of my question.

Edit (3x): in brief, see below and comments for more discussion:

Assuming the system clock follows Unix/POSIX time (not "real" UTC), how to get the real number of seconds elapsed since Unix epoch (from date or any other program)?

Or, at least, is there somewhere a "leap seconds" file in Linux where I can get the leap seconds without downloading them manually?

Explanation on how I arrived to the conclusion that my system clock is set to Unix time:

Reading the following lines from this link suggested by Richm:

As a result, for the most precise determination of epoch relative to the historic Gregorian calendar and UTC timescale, the user must subtract from the apparent NTP or POSIX epoch the relevant offset provided by the IERS.

Explanation on how I arrived to the conclusion that date --utc +%s really gives what we call "Unix timestamp":

The first leap second was introduced on June, 30th 1972.

Between 01 Jan 1970 00:00:00 (Unix epoch) and 01 Jul 1972 00:00:00, we can easily compute that there were 365 + 365 + (31+29+31+30+31+30) days + 1 leap second = 912 days + 1 leap second = 78796801 seconds. Now try date -R --utc -d @78796801... Output: Sat, 01 Jul 1972 00:00:01 !! If you thought (as I did before) that Unix timestamp directly gives the number of seconds elapsed in our real world since 01 Jan 1970 00:00:00... its wrong!

This example proves that date considers the value after the @ as a real Unix timestamp and gives the correct corresponding date following POSIX time definition. But how to use the same value and say that this is not a timestamp but a number of real seconds since the epoch?...

[Don't read the following: wrong initial assumption, I keep it for "memory"]

date -R --utc && date -R --utc -d @$(date --utc +%s)

Interpretation of the command line:

The first date gives the date set on my computer in UTC; the $(date ...) gives the Unix time, i.e. the number of seconds since Unix epoch minus the (25 this day) leap seconds; the date using this Unix time as parameter should then give a date 25 seconds in the past compared to the first command if it didn't manage properly the leap seconds. It's not the case, thus date must be "leap seconds-aware".

Community
  • 1
Yanux
  • 211
  • Your date command example does not prove anything about date being leap second aware. It just shows that %s and @ are consistent in their counting of seconds since the epoch. The full date manual page does state in relation to --utc that "Typically, system ignore leap seconds and thus implement an approximation to UTC rather than true UTC" – Richm Nov 05 '14 at 17:19
  • @Richm Yes, you're right. The problem was one step before: date doesn't know that my system is set to the "real" UTC (with leap seconds), thus date doesn't give me the Unix time but already Unix time + leap seconds... I realize then, that my question should be reformulated. I edit my question to clarify this point. – Yanux Nov 05 '14 at 17:45
  • 2
    I'm still not sure how you know that your clock is set to "real" UTC. For the most part Linux/Unix just side steps the existence of leap seconds and pretends that they never existed. Thus after a leap second the epoch actually references one second later than it did prior to the leap second. See if the description at https://www.eecis.udel.edu/~mills/leap.html helps. Specifically section 3 "How NTP and POSIX Reckon with Leap Seconds" which says "Thus, when a leap second is inserted in UTC and subsequently in NTP or POSIX, knowledge of all previous leap seconds is lost." – Richm Nov 05 '14 at 18:28
  • @Richm Well, I will read this carefully. But according to the Wikipedia link in my question, as far as I understand, POSIX time (the number itself) is not the same as UTC converted in seconds from the epoch, even if the two different values refer to the same moment. Anyway, now I don't know if my computer is set to real UTC or Unix time... I should open an auxiliary question about that maybe... – Yanux Nov 05 '14 at 18:43
  • @Richm Even if the sentence you quoted about POSIX forgetting all previous leap seconds sounds weird to me (as the text in your link reads: "there are as many NTP or POSIX timescales as historic leap seconds"!!!), I think you were right again and I edited my question according to your comment. Anyway, I am still waiting for a solution to automatically convert between real UTC and Unix timestamp, in either way... – Yanux Nov 06 '14 at 10:15
  • Your looking at a problem that doesn't exist IMHO. If you convert 2014-11-06T12:34:56Z to Unix epoch seconds you get 1415277296. If you display that with date you get 2014-11-06T12:34:56Z again. As someone else mentioned, Unix epoch seconds simply ignore leap seconds. So effectively there is a simple 1-to-1 mapping of Unix timestamps to UTC except for the leap seconds themselves. – wurtel Nov 06 '14 at 11:04
  • @wurtel I answer your comment in a new edit in my question. For me, it is a real problem, not obvious to solve, and there is no bijection between Unix timestamps and UTC because of the leap seconds precisely as you pointed out yourself. – Yanux Nov 06 '14 at 11:35
  • 1
    For reference I found the POSIX definition of seconds since the epoch at http://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap04.html#tag_04_14 This defines it as "A value that approximates the number of seconds that have elapsed since the Epoch. ... each and every day shall be accounted for by exactly 86400 seconds" – Richm Nov 06 '14 at 11:37
  • 1
    Take a look at the following on Stackoverflow it may help you to do what you want. http://stackoverflow.com/questions/19332902/extract-historic-leap-seconds-from-tzdata – Richm Nov 06 '14 at 11:38
  • You're adding the leap second for the 01 Jul 1972 example, but as Unix simply ignores leap seconds, you shouldn't! I still don't understand why the ignoring of the leap seconds is such a problem for you. Note that UTC doesn't have any concept of epoch and seconds since the epoch. – wurtel Nov 06 '14 at 12:05
  • @wurtel Ok, let's give a "real life" example: you have recorded an event with a device set to UTC and others with your computer adjusted to POSIX time, so you have corresponding Unix timestamps. You need to compare the events recorded with the device and on the computer. If you use UTC time, you need to know the leap seconds to convert it to the good timestamp. On the contrary, it should be okay if you put everything to UTC with date --utc -d @...: it's not an intuitive behavior even if it works. Anyway, a worse case: how do you compute accurate date differences with timestamps? – Yanux Nov 06 '14 at 13:48
  • @Richm Thank you for the links. I think there is no short solution to convert between "real number of seconds since the epoch" and "POSIX time". I think the best solution is to use this leap-seconds list and cope with that. I will try to write a short shell script using this file and post it there... – Yanux Nov 06 '14 at 16:44

1 Answers1

3

I didn't find a simple solution to my question, so I've written a small Bash script to solve it. You need to download the leap seconds file given in the link below and put it with the script or change the path to it. I didn't write utc2unix.sh yet, but it's very easy to adapt. Do not hesitate to comment/give suggestions...

unix2utc.sh:

#!/bin/bash

# Convert a Unix timestamp to the real number of seconds
# elapsed since the epoch.

# Note: this script only manage additional leap seconds

# Download leap-seconds.list from
# https://github.com/eggert/tz/blob/master/leap-seconds.list

# Get current timestamp if nothing is given as first param
if [ -z $1 ]; then
    posix_time=$(date --utc +%s)
else
    posix_time=$1
fi

# Get the time at which leap seconds were added
seconds_list=$(grep -v "^#" leap-seconds.list | cut -f 1 -d ' ')

# Find the last leap second (see the content of leap-seconds.list)
# 2208988800 seconds between 01-01-1900 and 01-01-1970:
leap_seconds=$(echo $seconds_list | \
               awk -v posix_time="$posix_time" \
               '{for (i=NF;i>0;i--)
                   if (($i-2208988800) < posix_time) {
                    print i-1; exit
                    }
                } END {if (($(i+1)-2208988800) == posix_time) 
                    print "Warning: POSIX time ambiguity:",
                            posix_time,
                          "matches 2 values in UTC time!",
                          "The smallest value is given." | "cat 1>&2"
                }')
# echo $leap_seconds

# Add the leap seconds to the timestamp
seconds_since_epoch=$(($posix_time + $leap_seconds))

echo $seconds_since_epoch

Just some tests:

  • date --utc +%s && ./unix2utc.sh -> today and at least until June 2015 the difference is 25 sec.
  • ./unix2utc.sh 78796799 -> 78796799
  • ./unix2utc.sh 78796801 -> 78796802
  • ./unix2utc.sh 78796800 -> 78796800 + on stderr: Warning: POSIX time ambiguity: 78796800 matches 2 consecutive values in UTC time! Only the smallest value is given.
Yanux
  • 211