How can I remove a carriage return, add a time stamp and ignore some data from a live MQTT feed

Question

I type mosquitto_sub -d -t +/# from the Ubuntu terminal to access the MQTT stream.

Real output from the live MQTT stream is this:

Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Received PUBLISH (d0, q0, r0, m0, 'm/gf/TMX6BP/075/d/SVlts', ... (28 bytes))
86,1224830,27610 27869 17565
Received PUBLISH (d0, q0, r0, m0, 'm/gf/TMX6BP/075/d/status', ... (39 bytes))
86,1243000,164573,-33.836465,151.051189
Sending PINGREQ
Received PINGRESP
Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/SVlts', ... (26 bytes))
806,3040421,7549 7750 3904
Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/status', ... (39 bytes))
806,3069000,59666,-33.836465,151.051189
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Sending PINGREQ
Received PINGRESP
Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/SVlts', ... (26 bytes))
810,5440995,6143 7807 4076
Sending PINGREQ
Received PINGRESP
Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/status', ... (39 bytes))
810,5489000,59897,-33.836465,151.051189
Sending PINGREQ
Received PINGRESP

There is no way of predicting when the next PUBLISH will be seen as they are only seen in the stream when the vehicle has transmission/reception from the GSM/3G towers

To filter I add mosquitto_sub -d -t +/# 2> >(grep PUBLISH) this will only allow lines with PUBLISH in it, hence the ouput is:

Received PUBLISH (d0, q0, r0, m0, 'm/gf/TMX6BP/075/d/status', ... (38 bytes))
86,637999,164563,-33.836465,151.051189
Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/SVlts', ... (26 bytes))
806,3040421,7549 7750 3904
Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/status', ... (39 bytes))
806,3069000,59666,-33.836465,151.051189
Received PUBLISH (d0, q0, r0, m0, 'm/gf/TMX6BP/075/d/SVlts', ... (28 bytes))
86,1224830,27610 27869 17565
Received PUBLISH (d0, q0, r0, m0, 'm/gf/TMX6BP/075/d/status', ... (39 bytes))
86,1243000,164573,-33.836465,151.051189
Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/SVlts', ... (26 bytes))
806,3640483,7463 7721 3933
Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/status', ... (39 bytes))
806,3674000,59676,-33.836465,151.051189
Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/SVlts', ... (26 bytes))
806,4240543,7291 7750 3933
Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/status', ... (39 bytes))
806,4279000,59687,-33.836465,151.051189
Received PUBLISH (d0, q0, r0, m0, 'm/gf/MXE/065/d/SVlts', ... (25 bytes))
455,24715,28041 28041 967

How would I be able to eliminate a few of the fields and also add a time stamp everytime I receive something; I 've tried using sed but had no luck. I entered $ mosquitto_sub -d -t +/# 2< <(grep PUBLISH) 2< <(sed "s/^/ date/") , $ mosquitto_sub -d -t +/# 2< <(grep PUBLISH) 2< <(sed "s/^/$date`/")

Q: How can I change my input to the terminal so that the output from the above live feed would be:

[timestamp],m,gf,TMX6BP,075,d,status,86,637999,164563,-33.836465,151.051189
[timestamp],m,NWRL,TMX,098,d,SVlts,806,3040421,7549 7750 3904
[timestamp],m,NWRL,TMX,098,d,status,806,3069000,59666,-33.836465,151.051189
[timestamp],m,gf,TMX6BP,075,d,SVlts,86,1224830,27610 27869 17565
[timestamp],m,gf,TMX6BP,075,d,status,86,1243000,164573,-33.836465,151.051189
[timestamp],m,NWRL,TMX,098,d,SVlts,806,3640483,7463 7721 3933
[timestamp],m,NWRL,TMX,098,d,status,806,3674000,59676,-33.836465,151.051189
[timestamp],m,NWRL,TMX,098,d,SVlts,806,4240543,7291 7750 3933
[timestamp],m,NWRL,TMX,098,d,status,806,4279000,59687,-33.836465,151.051189
[timestamp],m,gf,MXE,065,d,SVlts,455,24715,28041 28041 967

Possible solutions (future referencing):

Using the mosquitto_sub -d -t +/# 2> >(sed -n "s|.*$'.*',$.*|\1|p") | sed "N;s/\n/ /;s/$/ $(date)/" The output is:

0 810,5440995,6143 7807 4076 Wed Feb 25 23:23:51 UTC 2015 810,5489000,59897,-33.836465,151.051189 810,6041055,7606 7693 4076 Wed Feb 25 23:23:51 UTC 2015

Using the mosquitto_sub -d -t +/# 2> >(grep PUBLISH) | sed "N;s/\n/ /;s/$/ $(date)/" command from the terminal the output is

817,3069000,60045,-33.836465,151.051189 609,24570,27553 27553 955 Thu Feb 26 00:06:26 UTC 2015

Using the mosquitto_sub -d -t +/# 2>&1 | sed -n "/PUBLISH/{s|.*$'.*',$.*|\1|;N;s/\n/ /;s/$/ $(date)/;p}" The output is

'm/gf/MX3/122/d/status', 610,33000,28162,-33.836465,151.051189 Thu Feb 26 01:18:17 UTC 2015

Why is the terminal input relevant? Do you mean, what command should you type or is there reason a filter should read from the terminal itself to get this information rather than mosquito's stdout? Also, why do you think is the 86,6666...... appended line following a PUBLISH line in the second code block but not in the first one? Is a line like that not always going to follow a PUBLISH line? — mikeserv, Feb 25 '15 at 05:05
@don_crissti -wilco; ALAS! mosquitto_sub -d -t +/# 2>&1 | sed -n "/PUBLISH/{s|.*$'.*',$.*|\1|;N;s/\n/ /;s/$/ $(date)/;p}" has worked I just need to understand what does what now :/ and why the others didn't work? — 3kstc, Feb 26 '15 at 01:47

don_crissti · Accepted Answer · 2015-03-03T02:23:51.313

2

Well, rather confusing but anyway... Judging by the output of
mosquitto_sub -d -t +/# 2> >(grep PUBLISH)
your app seems to output to both stderr and stdout (otherwise you should only get lines matching PUBLISH in your output). It prints the debug messages (Sending... and Received...) to stderr and the actual data (810,5440995,6143...) to stdout.

Apparently, you need comma separated values so you could try the following, if you need the timestamp from the line matching PUBLISH:

mosquitto_sub -d -t +/# 2>&1 | xargs -d$'\n' -L1 sh -c 'date "+%s,$0"' | \
sed -n "/PUBLISH/{N;s|[ /]|,|g;s|^\([^,]*,\)[^']*'\([^']*\)',.*\n[^,]*,\(.*\)|\1\2,\3|;p}"

or, if you need the timestamp from the next line:

mosquitto_sub -d -t +/# 2>&1 | xargs -d$'\n' -L1 sh -c 'date "+%s,$0"' | \
sed -n "/PUBLISH/{N;s|[ /]|,|g;s|^[^,]*,[^']*'\([^']*\)',.*\n\([^,]*,\)\(.*\)|\2\1,\3|;p}"

2>&1 redirects stderr to stdout, the output is then piped to xargs which passes each line as an argument to the next command sh -c 'date "+%s,$0"' so each line is prepended with a timestamp+comma, e.g.:

[timestamp],Sending PINGREQ
[timestamp],Received PINGRESP
[timestamp],Received PUBLISH (d0, q0, r0, m0, 'm/NWRL/TMX/098/d/status', ... (39 bytes))
[timestamp],871,40114,4536 4536 323

This is then piped to sed suppressing the automatic printing (-n).
For each line matching PUBLISH, append the Next line, replace each space and / with comma then via grouping, retain only the first or second timestamp, the values in between quotes and the values after the second timestamp and finally, print the result:

[timestamp],m,NWRL,TMX,098,d,status,871,40114,4536,4536,323

edited Mar 03 '15 at 02:23

answered Feb 26 '15 at 02:29

don_crissti

82,805

I've looked into xargs I am understanding it a bit. How can I shift the the time stamp between before \1\2? ie [timestamp],m/NWRL/TMX/098/d/SVlts, 806,3040421,7549 7750 3904 – 3kstc Mar 02 '15 at 03:02
it works :( but it endlines the output here – 3kstc Mar 02 '15 at 03:51
you've got every right to be angry. but the guy responsible of me, is currently overseas, and we can't really communicate, so everything is a 2-3 days delayed. I will abide by you, and make it into a CSV file, I will look into it when I go home... don_cristi, thank you for your help thus far.. :) – 3kstc Mar 02 '15 at 05:16
there is one thing I want to do, and I will try to do it.. and if I can't, I promise you that will be the last question I will ask... but I think thank you is an understatement for the help and guidance you've given me. thank you! – 3kstc Mar 02 '15 at 05:29
white flags up, the terminal has defeated me! My last question is if you assist me with a minor tweek. And I will disappear! Many thanks! – 3kstc Mar 03 '15 at 00:54
it works! thank you! oh man, you rock my world! My first borns name: don_crissti, a tribute to you :) – 3kstc Mar 03 '15 at 02:02
I've looked at your code last night, and played around with it for a bit. I removed the $ from xargs -d$'\n' -L1 sh -c 'date.... so that its xargs -d'\n' -L1 sh -c 'date... and also I've removed the \ before the sed... it works great. What is the purpose of this two symbols? – 3kstc Mar 03 '15 at 23:12
I've embedded that line of code into a bash script. I am currently trying to figure out how to make it execute on startup/booting of the system. I figured via crontab I can type in @root /some directory to my script/script.txt and should work, and there is also the init.d way which I am currently looking into. – 3kstc Mar 04 '15 at 01:05
Very nice answer. Any simple way to pipe that output to a file? – Chuck Claunch Apr 06 '16 at 03:22
@don_crissti thanks for the quick reply! I tried your suggestion first and for whatever reason it doesn't actually write the file. If I just run the command by itself it works. I have write access and a simple echo 'test' > myfile works. I thought perhaps it was different with how xargs works or something (I never understood the magic voodoo that is xargs). – Chuck Claunch Apr 06 '16 at 20:27
I also put the command in a script and ran it with the > myfile to no avail. – Chuck Claunch Apr 06 '16 at 20:28
@don_crissti Yes, I'm adding it right to the end of the command. Here's a screenshot. I'm running Ubuntu 15.04 and bash. The MQTT stream is at 1 Hz and I verify that it prints to the screen at that rate. It's weird to me, seems like it should work. – Chuck Claunch Apr 06 '16 at 21:08
@don_crissti Interesting. I started it again and did a tail and didn't see anything so I let it run while I moved on to something else. Seems after about ten minutes it actually wrote the file and the tail spit out all of that data. The data is at roughly 1 Hz as expected. Any thoughts? Is there some kind of flush I could do on the file to force it to write every time? – Chuck Claunch Apr 06 '16 at 21:24
1

@ChuckClaunch - well, yeah, as I suspected, this is because of the pipe buffering - not much you can do other than use sed with -u (if you're using gnu sed that is) to force it unbuffer but that's all... – don_crissti Apr 06 '16 at 21:27
@don_crissti That works perfect! I just added that -u to the sed call and it works perfect. Thanks! – Chuck Claunch Apr 06 '16 at 21:30
1

@ChuckClaunch - no problem buddy, anytime ! – don_crissti Apr 06 '16 at 21:31

Walter A · Answer 2 · 2015-02-26T07:53:26.550

0

I simulated mosquitto in bash/ksh environment and found

stub_mosquitto | 
   sed -e 's/.*m0, //' -e 's/, .*bytes))//' | while read line; do
      if [[ "${line}" = \'* ]]; then
         echo -n "${line}, "
      else
         echo "${line}, $(date)"
      fi
   done

I used sed here for cutting the interesting parts.
I use while read line for a simple way of joining 2 lines (and adding a date). This is possible with sed or tr, but that might need some work to (different versions of sed).

I do not understand the call to mosquitto_sub exactly, so I only show the solution when the strings come through a pipe:

What is the # doing there?
Without quotes it should be the start of the comment
How do you get a line with numbers when you grep for a string?
Are the numbers written to stderr? After redirecting them to stdout with 2>&1 grep will skip them.

edited Feb 26 '15 at 07:53

answered Feb 25 '15 at 20:33

Walter A

736

would you be able to elaborate where and how I could implement this? (Apologies - totally new to Unix/programming) – 3kstc Feb 25 '15 at 22:27
Walter - why would you sed | read? – mikeserv Feb 26 '15 at 00:41
@mikeserv See updated answer with some explanation. – Walter A Feb 26 '15 at 07:54
It still doesn't make sense - do the join with sed - with read you only floor your efficiency and possibly mangle your input. – mikeserv Feb 26 '15 at 10:53
@mikeserv You are right. Joining lines with sed is much more efficient. I could have found the special syntax for removing \n on SO, I just tried to show a general approach in a "simple" way. – Walter A Feb 26 '15 at 21:41

How can I remove a carriage return, add a time stamp and ignore some data from a live MQTT feed

2 Answers2

Linked