I have a fortran program which I compiled myself and I ran the executable hundreds of times (without recompiling or anything), but now when I run it, it crashes instantly with segmentation fault. There are three other instances of the program running right now. top
outputs the following:
top - 15:37:06 up 5 days, 1:06, 2 users, load average: 3,00, 3,01, 3,06
Tasks: 290 total, 4 running, 285 sleeping, 0 stopped, 1 zombie
%Cpu(s): 24,4 us, 0,0 sy, 0,0 ni, 75,5 id, 0,1 wa, 0,0 hi, 0,0 si, 0,0 st
KiB Mem : 8058952 total, 2409096 free, 2964692 used, 2685164 buff/cache
KiB Swap: 8263676 total, 8263676 free, 0 used. 4614096 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1230 user 20 0 12,329g 675720 3080 R 100,0 8,4 14:17.45 tetramer
1236 user 20 0 12,329g 675688 3052 R 100,0 8,4 13:58.96 tetramer
1234 user 20 0 12,329g 675800 3168 R 100,0 8,4 14:02.23 tetramer
It does use a lot of memory (at least virtual memory), but up till now it used to be possible to run several instances at the same time as long as the actual memory usage was low enough. The fortran code in question follows, it crashes before getting to the write
.
IMPLICIT REAL*8(A-H,O-Z)
c
PARAMETER ( np = 220 )
c
PARAMETER ( ndim = 25000)
PARAMETER ( ndim2 = ndim*(ndim+1)/2 )
C
DIMENSION array(np,6,6),array2(np)
c
DIMENSION vector(50), vector2(50)
DIMENSION v1(159,30001),v2(159,30001),v3(159,30001)
C
COMMON /PARM/com1(99000) ,com2(0:8,0:8,99000)
1 ,com3(0:8),com4(0:8,0:8,0:8),nmax,mmax
1 ,com5(0:8,0:8)
C
COMMON /SET/ AX(0:4,-4:4,50),AY(0:4,-4:4,50),AZ(0:4,-4:4,50)
1 ,DD(0:4,-4:4,50), dd2(0:4,-4:4), nmax0(0:4,-4:4)
C
DIMENSION AH( ndim2 ),AF( ndim2 ),AF2(ndim2)
DIMENSION E( ndim ),VEC( ndim,ndim)
DIMENSION AH2(ndim,ndim),TEMP(ndim,ndim)
dimension nbarray(6)
C
CHARACTER*1 PARI
C
write(6,*) ' ###### ##### '
I really have no idea why I am getting a segmentation fault all of a sudden. As far as I can tell, I am not even accessing any memory in the program yet (just allocating), so how can I get a segmentation fault?
Also, when piping the output of the program into a perlskript, I got a SIGPIPE
for some reason, although it was the fortran program that crashed, not the perlskript.
Does anyone have any idea what might be happening here and how I can fix it?
I am running ubuntu 16.04 if that's relevant.
Edit: requested outputs are:
~$ ldd ./tetramer
not a dynamic executable
~$ strace ./tetramer
execve("./tetramer", ["./tetramer"], [/* 32 vars */]) = -1 ENOMEM (Cannot allocate memory)
--- SIGSEGV {si_signo=SIGSEGV, si_code=SI_KERNEL, si_addr=0} ---
+++ killed by SIGSEGV +++
Segmentation fault (core dumped)
I also did some testing and it's always the fourth instance of the program that crashes with segmentation fault. I recently did a reinstall (I wiped some older ubuntu and installed 16.04), and it may be that under 16.04 I could only ever run three at a time and didn't notice. The times where I am absolutely sure that there were more than three instances where all before the reinstall.
I think it may have to do with the fact that the program tries to allocate 12gb of memory when the total memory plus swap is only 16gb, but with the parameters I am using right now it only really needs around 1gb (in the RES column), so I don't see why I can't run more than three instances.
dmesg
, that's the only message today: do_general_protection: 33 callbacks suppressed (plus some messages about random time, whatever that is) – fifaltra Jun 13 '16 at 14:32ldd /path/to/your/fortran/program
. Then useapt-file
to find out which packages the libs belong to, and then something likezgrep ' upgrade PACKAGENAME ' /var/log/dpkg.log*
to find out when they were last upgraded. Also run your program withstrace
- that will tell you exactly where it is crashing...add the last 10 or 20 lines of the strace output to your question. – cas Jun 14 '16 at 07:03