Uploading a binary file with the REST API

[Interferon]

I am trying to upload a 4.18 MB ZIP file using the /downloads/files REST API endpoint.

If I add the file contents as binary data, the uploaded file is five bytes. This corresponds with the zip data, which has 0 at the sixth byte:

80
75
3
4
20
0
0
0

So the NULL character is causing the string to get cut off prematurely.

I can upload the data with base64 encoding, and it results in a 5.58 MB file. When I download this file, then run it through base64 decoding, it produces the original ZIP file. However, the base64 encoded data is not a valid zip file.

My question is how do I upload a binary file, with or without base64 encoding, and have IPB store it as the actual binary data?

WString path = GetPath(PATH_DESKTOP) + "/Bamboo001A.zip";
auto stream = ReadFile(path);
    
//Base64 version
//auto buffer = CreateBuffer(stream->GetSize());
//stream->Read(buffer->Data(), stream->GetSize());
//std::string s = base64_encode((const unsigned char*)buffer->Data(), buffer->GetSize());
    
//Binary version
std::string s;
s.resize(stream->GetSize());
stream->Read(s.data(), s.size());

for (int n = 0; n < 8; ++n) Print(s[n]);

std::string token = InvisionPower::AuthenticateUser(name, password, scopename);
std::string options = ("category=97&description=MyDescription&title=" + StripAll(path).ToUtf8String() + "&author=1&");
    
options += "files[" + StripDir(path).ToUtf8String() + "]=" + s;
auto j3 = InvisionPower::APIPost("/downloads/files/", token, options);
    
if (j3["response"].is_string()) Print(std::string(j3["response"]));

 

Edited October 5 by Interferon

[Interferon]

I found that it works if you read the binary file data into a string and then do a url-encode operation. This is strange to me because I never associated that with binary data before, but it works.

Also, make sure your php.ini file allows file sizes as big as you are trying to upload, and gives the uploader enough time to finish. For a max file size of 1 GB and 10 minutes upload time, the settings would be like this, I think:

upload_max_filesize = 1024M
post_max_size = 1024M
memory_limit = 1024M
max_input_time = 600
max_execution_time = 600

You must restart your server after making changes to PHP settings.

If your upload fails you will not receive any meaningful error codes. In my case it just says "no categeory" (sic), probably because the data is getting prematurely terminated and the remaining fragment isn't parsed correctly. Perhaps the REST API should accept an optional filesize[] parameter so that an error message can be displayed if the received file data does not match the expected value?

It's best to test with very small files to begin with, and then scale up and debug as you hit problems uploading bigger files, and just try uploading files without screenshots to start with.

[Interferon]

Unfortunately, I cannot upload files bigger than a few megabytes, and any attempt to include a screenshot using the method above results in the "no categeory" error, even using extremely small files, like a 10x10 PNG with a 6-byte text file.

The same files upload without any issues in the web interface.

Edited October 7 by Interferon

[Interferon]

I was able to upload very large files (240MB) with a screenshot without issue by uploading the files to our server and running a PHP script. It appears there is some critical piece neither the web devs nor the desktop developers understand.

<?php

$apiKey = "xxxxxxxxxxxxxxxxxxxxxxxxxx";
$category = 97;
$author = 1;
$title = "Bamboo001A";
$description = "Bamboo001A_1K by AmbientCG. This file was uploaded via an automated system.";

// API endpoint
$url = "https://www.xxxxxxxxxxxxxxxxxxxxxxxxxx.com/api/downloads/files";

// Prepare the file upload
// $cfile = new CURLFile($filePath);
$data = [
    'key' => $apiKey,
    'files[Bamboo001A_1K-PNG.zip]' => file_get_contents("Bamboo001A_1K-PNG.zip"),
    'files[Bamboo001A_2K-PNG.zip]' => file_get_contents("Bamboo001A_2K-PNG.zip"),
    'files[Bamboo001A_4K-PNG.zip]' => file_get_contents("Bamboo001A_4K-PNG.zip"),
    'screenshots[Bamboo001A.jpg]' => file_get_contents("Bamboo001A.jpg"),
    'category' => $category,
    'author' => $author,
    'title' => $title,
    'description' => $description,
];

// Initialize CURL
$ch = curl_init();

// Set CURL options
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);

// Execute the request
$response = curl_exec($ch);

// Check for errors
if (curl_errno($ch)) {
    echo 'Error: ' . curl_error($ch);
} else {
    // Handle the response
    echo 'Response: ' . $response;
}

// Close CURL
curl_close($ch);
?>

 

[Interferon]

I also tried adding the post field size explicitly, but it made no difference, with or without file data url-encoded.

if (!postData.empty())
{
  	curl_easy_setopt(curl, CURLOPT_POSTFIELDSIZE, postData.size());
	curl_easy_setopt(curl, CURLOPT_POSTFIELDS, postData.c_str());
}

 

Edited October 7 by Interferon