[SOLVED] Can you read and delete chunk of a file?

Issue

I am using ofstream and ifstream to read a chunk from a file post it over middleware (DDS) to another process and the other process writes that chuck of the file.

Basically transferring a file. The two components are unaware of each other and may live on the same hardware or on different hardware (DDS takes care of the transfer either way).

This is all working, however when I try to do this with a large file (>500MB) and if the destination component is on the same board, then I run out of RAM (since 500 x 2 = 1GB which is my limit).

So I am thinking of reading a chunk from a file deleting that chunk of the file and then sending the chunk. So I end up with:

A        B
12345 ->
2345  -> 1
345   -> 12
45    -> 123
5     -> 1234
      -> 12345

Where each number is a chunk of a file.

I am using linux, so I can use any linux APIs directly, but would probably prefer a pure c++ approach. I can't really see any good options here. i/ostream does not appear to let you do this. Options like sed will (I think) end up using more memory by copying.

Are there any better mechanisms for doing this?

Update

The files are stored in RAM via a tmpfs partition

Solution

I am using linux, so I can use any linux APIs directly, but would probably prefer a pure c++ approach. I can't really see any good options here. i/ostream does not appear to let you do this. Options like sed will (I think) end up using more memory by copying.

Are there any better mechanisms for doing this?

There is no standard mechanism in C++ or Linux for shortening a file in-place by deleting data from the beginning or middle. Most file systems don't work in a way that would support it. When one wants to delete data from such a position, one has to make a new copy of the file, omitting the data that are to be deleted.

You can shorten a file by removing a tail, but that does not serve your purpose unless possibly if you send chunks in reverse order, from tail to head. However, the most natural ways I can think of to support that in an application such as yours would involve pre-allocating the full-size destination file, and that would have the same problem you are trying to solve.

Answered By - John Bollinger

Answer Checked By - David Goodson (WPSolving Volunteer)