4

I am executing some benchmark using a DBMS and virtual machines. And I am using this command

free && sync && echo 3 > /proc/sys/vm/drop_caches && free

on each Guest, and on each Host, before executing a query, to minimize the effect of cache/buffering on my execution time. If the data is cached/buffered the execution time for a query may vary depending on what query was executed before, because a part of the file used to answer the query might be in memory.

I saw this post: How do you empty the buffers and cache on a Linux system?

But for me it's a little tricky to execute sync/drop caches after each query. I need to ask for every VM and for every physical server to sync/drop caches. I could write a script, and use SSH to do this, but I wonder if there's a way to keep the OS from buffering/caching the files because it would be easier for me.

I am using Ubuntu 12.04 on all the VMs (using KVM) and servers.

Community
  • 1
hlustosa
  • 141
  • It doesn't really make sense to completely disable caching. This would mean that every instruction that executes in a program would have to be fetched from disk, instead of loading the program into memory in whole blocks. – Barmar Oct 15 '14 at 18:15
  • Is there a way to avoid caching for files used by a specific process? All I want is to garantee that the queries will cause the data to be fetched from the disk, and not from memory. – hlustosa Oct 15 '14 at 18:24
  • So if the program reads from the file a byte at a time, each read should go to disk instead of using the block that was read for the previous read? I suspect that's not what you want, either. You just want the cache to be flushed before the program starts. – Barmar Oct 15 '14 at 18:26
  • Note that anything you do regarding the VM's cache probably won't affect the host's cache. – Barmar Oct 15 '14 at 18:27
  • @Barmar I execute queries for my distributed dbms to understand what is the better way to partitionate the data. The dbms retrieves blocks of data at time. I execute a series of queries, and I need to repeat the execution of the same query many times in a row. If the blocks are in memory, the execution time will be slower, and I can't use it to compare with another query. I could restart the whole system everytime, but it would take me forever. – hlustosa Oct 15 '14 at 18:41
  • When I flush the cache, I do this for the VM and for each Host I use. I assume that some o blocks of the VDI file might be in memory while the hypervisor is reading/writing on it. – hlustosa Oct 15 '14 at 18:43
  • If you're getting “strange results”, it means your setup is broken. You've given so little information (not even the VM technology!) that we can't help you repair it. – Gilles 'SO- stop being evil' Oct 15 '14 at 23:15
  • 2
    If sync fixes the problem, then they aren't due to caching, but to buffering. – Gilles 'SO- stop being evil' Oct 15 '14 at 23:16
  • This is the command that solves my problem: free && sync && echo 3 > /proc/sys/vm/drop_caches && free

    I don't actually get strange results. The results I am getting are expected, but I want to minimize the effect of caching/buffering the file system data. I am using KVM as my hypervisor. Both guests and hosts are using ubuntu server.

     – hlustosa Oct 16 '14 at 08:45

1 Answers1

0

Use something like https://code.google.com/p/linux-ftools/ to mark all the files used by the database system as FADV_DONTNEED?

Josip Rodin
  • 1,138