Comparison in IFILE_Read: char by char Vs chunk of memory | developer.brewmp.com Comparison in IFILE_Read: char by char Vs chunk of memory | developer.brewmp.com

Developer

Comparison in IFILE_Read: char by char Vs chunk of memory

Forums:

Greetings!

I want to parse and read data from a file, line by line.

I can think of two approaches:

1) Read data char by char, and check if '\n' is reached
2) Read chunk of data at a time (example: 256 bytes), then parse the data and fill in memory buffer accordingly

The former approach appears better, resulting in optimal code (wrt my algo i've thought of).

My question is: Is reading from a file character by character slower process than reading chunk of data one time then process it?

If so, then is there a very big difference or is it negligible?

All inputs are appretiated!

Thanks...

Taha

2) Read chunk of data at a time (example: 256 bytes), then parse the data and fill in memory buffer accordingly
Would give you better result, file I/O is more expensive, more specifically in the deivce.

2) Read chunk of data at a time (example: 256 bytes), then parse the data and fill in memory buffer accordingly
Would give you better result, file I/O is more expensive, more specifically in the deivce.

taha wrote:My question is: Is reading from a file character by character slower process than reading chunk of data one time then process it?
Usually.
taha wrote:If so, then is there a very big difference or is it negligible?
It shouldn't be too hard to write a simple benchmark and measure it. It's possible that the speed of flash combined with however the BREW file system works makes in negligible, but unless someone has measured it you can't know for sure.

taha wrote:My question is: Is reading from a file character by character slower process than reading chunk of data one time then process it?
Usually.
taha wrote:If so, then is there a very big difference or is it negligible?
It shouldn't be too hard to write a simple benchmark and measure it. It's possible that the speed of flash combined with however the BREW file system works makes in negligible, but unless someone has measured it you can't know for sure.

For optimal speed, you should read in chunks. We have seen dramatic improvements in on device performance by buffering our read / writes. In general, it is best to limit these to as few as possible.

For optimal speed, you should read in chunks. We have seen dramatic improvements in on device performance by buffering our read / writes. In general, it is best to limit these to as few as possible.

I believe the optimal size for file I/O is in multiples of 4 KB -- I recall hearing at some point that AMSS uses 4 KB buffers, so if you constrain yourself to file reads/writes that fill up the whole buffer, you'll improve throughput.

I believe the optimal size for file I/O is in multiples of 4 KB -- I recall hearing at some point that AMSS uses 4 KB buffers, so if you constrain yourself to file reads/writes that fill up the whole buffer, you'll improve throughput.

rabidcow wrote:Usually.
It shouldn't be too hard to write a simple benchmark and measure it. It's possible that the speed of flash combined with however the BREW file system works makes in negligible, but unless someone has measured it you can't know for sure.
I ran this test a couple of weeks ago. I made it about 30 times quicker to read 10,000 bytes in one go versus doing 10,000 single byte reads.
Test was run on a V3C. It does not appear to do anything with IFILE_SetCacheSize. I didn't do any further testing - e.g. is there an optimum size.
An alternative would be to write a caching replacement for IFile which does implement IFILE_SetCacheSize - and then use that throughout your application. That wouldn't to be hard either.

rabidcow wrote:Usually.
It shouldn't be too hard to write a simple benchmark and measure it. It's possible that the speed of flash combined with however the BREW file system works makes in negligible, but unless someone has measured it you can't know for sure.
I ran this test a couple of weeks ago. I made it about 30 times quicker to read 10,000 bytes in one go versus doing 10,000 single byte reads.
Test was run on a V3C. It does not appear to do anything with IFILE_SetCacheSize. I didn't do any further testing - e.g. is there an optimum size.
An alternative would be to write a caching replacement for IFile which does implement IFILE_SetCacheSize - and then use that throughout your application. That wouldn't to be hard either.

Greetings!
All inputs received here have been extremely helpful in analysis between both approaches. I have some observations:
------------------------------------------------------------------------------------------------
there was a 3171 kb FILE
with 4096 bytes per read function, read time was 23.61 secs
with 256 bytes per read function, read time was 363.796 secs (about 6 mins)
so former approach was 15 times faster
with 8192 bytes per read function, read time was 12.266 secs
with 10000 bytes per read function, read time was 9.437 secs
between 8192 vs 4096, read time has increased 1.9 times
between 10000 vs 4096, read time has increased 2.5 times
char by char is nowhere near competition.
wonder how much would actual device allow easily with optimal performance. because for the simulator, it turns out that the more data read at a time (within
some upper bounds), the better. for the device however it is required to let the device and its watchdog do other processing. I'm assuming mohlendo's way
should work better 'on' the device?
p.s. for a file of about 10 MB size, application crashed about after reading about 4-5 MB of data - interesting
------------------------------------------------------------------------------------------------
Regards,
Taha

Greetings!
All inputs received here have been extremely helpful in analysis between both approaches. I have some observations:
------------------------------------------------------------------------------------------------
there was a 3171 kb FILE
with 4096 bytes per read function, read time was 23.61 secs
with 256 bytes per read function, read time was 363.796 secs (about 6 mins)
so former approach was 15 times faster
with 8192 bytes per read function, read time was 12.266 secs
with 10000 bytes per read function, read time was 9.437 secs
between 8192 vs 4096, read time has increased 1.9 times
between 10000 vs 4096, read time has increased 2.5 times
char by char is nowhere near competition.
wonder how much would actual device allow easily with optimal performance. because for the simulator, it turns out that the more data read at a time (within
some upper bounds), the better. for the device however it is required to let the device and its watchdog do other processing. I'm assuming mohlendo's way
should work better 'on' the device?
p.s. for a file of about 10 MB size, application crashed about after reading about 4-5 MB of data - interesting
------------------------------------------------------------------------------------------------
Regards,
Taha

Useful results - but you really need to run this on device. Simulator values are just meaningless, as the characteristics of a hard disk with a substantial cache and a flash chip are not comparable.

Useful results - but you really need to run this on device. Simulator values are just meaningless, as the characteristics of a hard disk with a substantial cache and a flash chip are not comparable.