Placement of downloaded files / downoaded files with inaccurate file sizes

[??????]

Hey there...

I'm really not sure if anyone is able to help me, or if anyone here even has experience with similar situation, but I decided to ask anyway...

I have scripts in place for allowing me to process HTTP requests.  I have written an automatic updating system to accompany the aforementioned HTTP request scripts.  What the updating system does is send a simple get request to my server with the path of the file to download.  Normally, this is enough to trigger the initial 'download' of the file in question; however, this does not allow me to specify where to save the file to.

So, I was faced with the issue of 'ensuring downloaded files are placed into their correct folders'. I got around this by making a simple method similar to below...

    # from_dir_and_name  - specifies the server path for the file to download.    # to_dir_and_name    - specifies the game folder path to save the file to. def download_file(from_dir_and_name, to_dir_and_name) get(from_dir_and_name) do |reply| # Proccess the request reply - ie, the file.. File.open(to_dir_and_name,'wb') { |file| file.write(reply) } end endAnd guess what, it worked!!  However, for some reason the files I download are a *tiny* bit larger than the files stored on my server.

Example: I have 'FilenameABC', which is 1000 bytes exactly.  When the game downloads this file into its designated folder, it will have increased in size by as much as 7 bytes.

Now, you may be thinking '7 bytes', its hardy different... But in my eyes, its very different.  The actual file contents seems to be identical, but I haven't ran any extensive tests on them yet. I also have 'no eyed deer' what the cause of these extra byes could be.

So yea, there's a bit of 'backstory' for my post.

My main issue is that - in order for me to save the file where i desire, i must write the reply onto another file, which is creating excess bytes.

Is anyone able to provide any insight on what could cause this, or an alternative method for me to direct the file towards a designated path (into MyKewelFolder for example)

Many thanks in advance.

 

[Tsukihime]

Just run a diff to see what the new things are. It's only 1000 bytes.

Any number of factors could result in this, including things like encoding conversion.

def download_file(from_dir_and_name, to_dir_and_name) get(from_dir_and_name) do |reply| # Proccess the request reply - ie, the file.. File.open(to_dir_and_name,'wb') { |file| file.write(reply) } end end

What is the size of your reply?

 

[??????]

Dunno, didnt check the size of the reply, but i assume its the same as the file size as the request is just getting a file directly, its not running through a php request.

So like, the get request would be similar to this...

get("hostname/pathname/filename.rvdata2")so the reply is the rvdata2 file itself.

 

[Tsukihime]

but i assume

Errors occur because assumptions are wrong.

 

[??????]

Thats a valid point.  When I relaunch the server I will check the size of the reply

However, I think you may have been onto something with the encoding of the data.  :/

 

[Tsukihime]

Those bytes can be the size of the response as well.


Depending on what library you're using to make requests that will be added so the client knows how many bytes to read.

 

[??????]

Ok, I checked that stuff out last night...

The reply size is equal to the size of the file from the server - ie, the correct file information.  So the reply is returning the accurate file 'string' but when its getting written onto the new file, its still gaining the new bytes. So yea, my assumption there as accurate ~ cant lie, I'm pleased about that.

All that im writing onto the new file is the reply (shown below, again), so i really don't know why its being so strange...

    def download_file(from_dir_and_name, to_dir_and_name)      HTTP::GAME.send('get',from_dir_and_name) do |reply|        File.open(to_dir_and_name,'wb') { |file| file << reply }      end    endNote: I also tried to write each byte t the new file individually like so...

File.open(to_dir_and_name,'wb') do |file| reply.each_byte { |byte| file << byte } endBut that just yielded insane results (file sizes increasing in size x3) which makes sense cause its adding the individual bytes

 

[Galenmereth]

It could just be the way Windows calculates filesize vs how linux calculates filesize; I assume your server is some *nix system?

 

[??????]

Afraid not ~ been testing on my 'private server' which is just 'abyss web server' on my windows laptop

I also tried sending some png's, which gave funny results.. It is sending the file, and i can open it as a png, and it shows imagery, but only a portion of the top of the image. :/

 

[Tsukihime]

It seems you are satisfied with the output and there is nothing wrong since it doesn't affect your program.


Otherwise, you will either need to upload the original file and the resulting file, or you need to run a diff and look at what those extra bytes (if any) are.


It could simply be a case of Windows explorer not getting the filesize right for whatever reason.

But that just yielded insane results (file sizes increasing in size x3) which makes sense cause its adding the individual bytes

This doesn't make sense to me: what is the difference between writing one byte out at a time, and writing the entire data stream out?

 

[??????]

Yes, this is a major issue as iles are not correct when they download, yet I have not found what the actual difference is.  text files seem to download accurately, rvfiles do not, and png's download the file in the correct size, but only display about 4-8 pixels of the top of the image.

Yea, I realized it would be best to upload some files, so i made an example txt file, rvdata 2 file and png file.

Link to files :: https://www.dropbox.com/sh/o9kaqvm2lblk3zj/AABdcit5QLUiDNz481SuTJvGa?dl=0

The folder linked above contains each of the files in both 'normal' form and also in 'downloaded' form.

This doesn't make sense to me: what is the difference between writing one byte out at a time, and writing the entire data stream out?

A byte of information is different from a character. If writing a byte, you are writing the characters that the byte would display. Eg...

"ABC".each_byte.with_index do |byte,index| puts "Byte @ #{index} == #{byte}"end# Output : # Byte @ 0 == 65# Byte @ 1 == 66# Byte @ 2 == 67Hope that helps

Edit:

To get the actual character of the byte just have to .chr it.

"ABC".each_byte.with_index do |byte,index| puts "Byte @ #{index} == #{byte.chr}"end# Output : # Byte @ 0 == A# Byte @ 1 == B# Byte @ 2 == CFurther edit:

I'm awesome.

The reason that part of the files where missing was due to a silly miscalculation in my http script that wasnt allocating enough buffer to hold the full file. That is now fixed; however, the file sizes are still slightly different. ~ I updated the files in the above download to the new ones.

Now the files are actually working though, so thats most of the battle, I just have to figure out what these extra bytes are :/

then ill get a way to calculate how much buffer is needed for each file download. before it tries to download it.

 

[Tsukihime]

There's a bunch of spaces at the end of your rvdata file.


Next time look at your files: it is not difficult to run a diff.

A byte of information is different from a character. If writing a byte, you are writing the characters that the byte would display. Eg...

Yes, but you should not be writing bytes as direct literals.


10 is not the same as 0x10


It doesn't make sense to try to compare the two, so I didn't think you actually meant that.

 

[??????]

Oh, nice catch on the spaces, never spotted those.

I would .strip it, but that causes issues when downloading image type files.  Any suggestions on what I could do there?

Do you think it could be a case of allocating too big a buffer?

Edit:

Ok, I tried to just .rstrip the reply string, which worked wonders.  png files have some utf8 error when you try to .lstrip or .strip the reply string.

Anyway, I used .rstrip on the reply and it seems to have heavily reduced the differences, I did however then have to add a null character at the end of the reply string (shown below) ~ I just hope that doesn't raise an issue further down the line with some alternate file type :/

File.open(to_dir_and_name,'wb') do |file| file << reply.rstrip << "\0" endThere still seems to be a small difference in the png file though. Its now only 1byte larger, but still, a byte is a byte

I re-updated the files in the dropbox linked above.

 

[Tsukihime]

Try allocating a buffer twice the size and see what hppens

 

[??????]

Oh, didnt see your post when i was editing my previous post

I tried increasing the buffer by another  ' ' * 1024, nothing different occurred.

 

[Tsukihime]

You need to figure out where or why the spaces are added.


This isn't a matter of size, it's a matter of integrity.


If you released an HTTP library and it appended random spaces or null characters to requested data, that would be considered incorrect transfer and the client SHOULD ask the server to re-send the packet.


If my program performed CRC checks it will likely fail.

 

[??????]

Agreed,

I already decided to rewrite the core of my http library, so hopefully something stands out when im doing that.  But as of right now, im looking over the files in notepads and trying to compare them for any differences other than spaces.  Cant see any real difference other than that though