1 Topic by Daniel

  • Daniel
  • Guest

Topic: Splitting the log file

Hi,

We ran a test for 5 days, 700Mb log file, but even a 2GB RAM machine can't open it ("Out of Memory" error).  There's at least 3 million records.  Please, how can we split the binary log file we have?  We really can't afford to throw away 5 days of testing!

(yes, we've already set it to hourly splits for future tests)

thanks,
Daniel

2 Reply by Daniel

  • Daniel
  • Guest

Re: Splitting the log file

Never mind.. we used the known format of the file to write a simple splitter application.  Thanks!

3 Reply by tgurock

  • tgurock
  • Administrator
  • Offline
  • Registered: 2009-07-09
  • Posts: 628

Re: Splitting the log file

Hello Daniel,

Thanks for your interest in SmartInspect.

Writing a tool for splitting the log file should indeed be the best way to handle this. Such a tool should normally require less than 100 lines of code. Please note that there's no need to fully read and extract all properties of the individual packets. It is sufficient to read the packet header, look at the packet size, read the packet body and then write the entire packet to an output file. In pseudo code this could look like: 

MAXIMUM_SIZE = 50 * 1024 * 1024  { 50 MB }

OpenInput()
try
  if (!ReadSILF()) then { Read log file header }
  begin
    Error("Not a SmartInspect log file");
  end

  OpenOutput()
  try
    WriteSILF() { Write log file header }

    currentSize = 0
    while ReadHeader(out header) do
    begin
      if currentSize + header.Size > MAXIMUM_SIZE then
      begin
        SwitchOutput() { Open the next output file }
        currentSize = 0
      end
      currentSize += header.Size
      if ReadBody(header, out body) then
      begin
        WritePacket(header, body)
      end
    end
  finally
    CloseOutput()
  end
finally
  CloseInput()
end

Please let me know if you have any questions. You can also contact me via email (simply click on the my name below).