Issue

I wrote a small program to collect data over serial port using MinimalModbus. The data is dumped into a CSV file. I have read several posts on SO and other places. A few things mentioned are:

Using lazy evaluation wherever possible (xrange instead of range)
Deleting large unused objects
Use child processes and upon their death memory is released by OS

The script is on github here. I also use a script to periodically upload these files to a server. Both these scripts are fairly trivial. Also nothing else is running on the system, thus i feel that withing these two systems only memory hogging is taking place. What would be the best way to tackle this issue. I am not the most willing to adopt the subprocess route.

Some more information:

Data collection is on Raspberry Pi (512 MB RAM)
Python version: 2.7
It takes about 3-4 days for RAM to be completely used after which the RaspberryPi freezes

I followed this guide to find out top 20 programs which are eating up RAM.

$ ps aux | awk '{print $2, $4, $11}' | sort -k2rn | head -n 20
12434 2.2 python
12338 1.2 python
2578 0.8 /usr/sbin/console-kit-daemon
30259 0.7 sshd:
30283 0.7 -bash
1772 0.6 /usr/sbin/rsyslogd
2645 0.6 /usr/lib/policykit-1/polkitd
2146 0.5 dhclient
1911 0.4 /usr/sbin/ntpd
12337 0.3 sudo
12433 0.3 sudo
1981 0.3 sudo
30280 0.3 sshd:
154 0.2 udevd
16994 0.2 /usr/sbin/sshd
17006 0.2 ps
1875 0.2 /usr/bin/dbus-daemon
278 0.2 udevd
290 0.2 udevd
1 0.1 init

So the two Python processes are eating up some RAM, but that is very small when compared to overall RAM consumed. The following is the output of the free command.

pi@raspberrypi ~ $ free -m
             total       used       free     shared    buffers     cached
Mem:           438        414         23          0         45        320
-/+ buffers/cache:         48        389
Swap:           99          0         99

The following is the output of the top command.

Tasks:  69 total,   1 running,  68 sleeping,   0 stopped,   0 zombie
%Cpu(s): 66.9 us,  5.0 sy,  0.0 ni, 18.1 id,  0.0 wa,  0.0 hi, 10.0 si,  0.0 st
KiB Mem:    448776 total,   429160 used,    19616 free,    47016 buffers
KiB Swap:   102396 total,        0 used,   102396 free,   332288 cached

PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND           
12338 root    20   0 10156 5644 2384 S  69.8  1.3   3059:31 python            
26039 root    20   0     0    0    0 S   1.6  0.0   0:02.71 kworker/0:1       
26863 pi      20   0  4664 1356 1028 R   1.3  0.3   0:00.12 top               
1982 root     20   0  1752  516  448 S   0.3  0.1   1:08.36 sh                
1985 root     20   0  1692  552  460 S   0.3  0.1   5:15.16 startpar          
1 root        20   0  2144  728  620 S   0.0  0.2   0:17.43 init              
2 root        20   0     0    0    0 S   0.0  0.0   0:00.14 kthreadd          
3 root        20   0     0    0    0 S   0.0  0.0   0:13.20 ksoftirqd/0       
5 root         0 -20     0    0    0 S   0.0  0.0   0:00.00 kworker/0:0H      
7 root         0 -20     0    0    0 S   0.0  0.0   0:00.00 kworker/u:0H      
8 root         0 -20     0    0    0 S   0.0  0.0   0:00.00 khelper           
9 root        20   0     0    0    0 S   0.0  0.0   0:00.00 kdevtmpfs         
10 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 netns             
12 root      20   0     0    0    0 S   0.0  0.0   0:00.06 bdi-default       
13 root       0 -20     0    0    0 S   0.0  0.0   0:00.00 kblockd

EDIT 2

As suggested in the first answer, i decided to look into log files. I had a look at syslog and the following is the result of tail on it.

May 19 10:03:26 raspberrypi wpa_supplicant[7065]: wlan0: Failed to initialize driver    interface
May 19 10:03:49 raspberrypi wpa_supplicant[7157]: nl80211: 'nl80211' generic netlink not found
May 19 10:03:49 raspberrypi wpa_supplicant[7157]: Failed to initialize driver 'nl80211'
May 19 10:03:49 raspberrypi wpa_supplicant[7157]: rfkill: Cannot open RFKILL control device
May 19 10:03:49 raspberrypi wpa_supplicant[7157]: Could not read interface wlan0 flags: No such device

These messages are filling up the log files and are coming every second. The interesting part is that i am using Ethernet and not WiFi.

Thus, now it is unclear where the RAM has gone?

Solution

Most of your RAM is free for applications, because it's used for the buffers and caching. Look at the "-/+ buffers/cache:" line to see the amount of RAM that is really used/free. An explanation can be found here.

To verify wether Python is leaking memory, monitor that python's RSS size (or %mem) over time. E.g. write a shell-script that is called from a cron job every couple of hours to append the output of your ps command chain and the output of the free command to a file.

If you find that the Python processes are leaking memory there are a couple of things you can do;

Modify your script to that it extis after 24 hours and use a e.g. a cron job to restart it (the easy way out.)
Take an in-depth look into Python itself and expecially into the extension modules you're using. Use the gc module to monitor and influence the memory usage. You can e.g. call gc.count() regularly to monitor the amount of objects marked for collection. You can call gc.collect() explicitly and see if that reduces memory usage. You could also modify the collection threshhold.

If Python's RAM use doesn't increase over time, it could be another program of daemon. The memory logging script that I mentioned above should tell you which one it is.

There could also be another reason that your computer freezes. Look at the Linux logfiles for clues.

Edit: Since you have wpa_supplicant filling up the log file, you should check the state of the filesystem(s). A full filesystem might cause the system to hang. If you aren't using the wireless interface, disable it.

Answered By - Roland Smith

Answer Checked By - Senaida (WPSolving Volunteer)

Friday, February 4, 2022

[SOLVED] Python program eating up RAM

Issue

Solution