Thursday, February 3, 2022

[SOLVED] Python: OverflowError long int too large to convert to int

Issue

After many years finding answers here, it's finally my time to ask my first question ever.

Running a RPI3B+ like a mini-server at home, I want to ba able to send large files to friends and family over internet, via CLI. To do this, I am using this : https://github.com/justbeamit/beam/blob/master/beam

But, when I want to upload a file larger than ~1.5Gb (by my estimate), I get an error saying :

OverflowError
long int too large to convert to int

After a short investigation, I can see that it comes from the line 288 where the max value of the progress bar is set, in this method :

def transfer(token, filePaths):

  print("recipient has connected! starting transfer...")

  uploadUrl = ACTIVE_BACKEND + "/upload"

  try:

    index = 0
    ProgressBar.totalNumberOfFiles = len(filePaths)

    for filePath in filePaths:

      # make the console look pretty
      sys.stdout.write("\n")
      print("  " + filePath)

      # the callback function invoked by the monitor
      def updateProgress(monitor):
        theProgressBar.update(monitor.bytes_read)


      # setup the multi-part encoder & its monitor
      fileMPE = getMultipartEncoder(filePath)
      monitor = MultipartEncoderMonitor(fileMPE, updateProgress)

      # setup the progress bar
      ProgressBar.fileNumber = index + 1 # to avoid showing (0 of 3)

      # since the progress bar will be updated by the multi-part encoder, we can't set 'maxval'
      # to be the file's size since the encoder adds extra bytes to account for the header
      theProgressBar = ProgressBar(
        maxval = len(fileMPE), 
        widgets = WIDGETS,
      )

      theProgressBar.start()

      urlParams = {
        "type": "CLI",
        "token": token,
        "index": index
      }

      requests.post(
        uploadUrl,
        data=monitor,
        params=urlParams,
        headers={'Content-Type': monitor.content_type}
      )

      theProgressBar.finish()
      index += 1
    # end for loop

  except Exception as e:
    print(e.__class__.__name__)
    print(e)
    exit()

  finally:
    sys.stdout.write("\n")

Can anyone help me? This is annoying because without the progress bar, everything would work perfectly fine. I tried commenting this line, but then the error moves somewhere else, at line 300 (requests.post).

My infos :

python --version => Python 2.7.13

Raspbian version => PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)"


Solution

Are you sure the combined file size is not larger than 4 GB? I think you're hitting this Python "bug" as your Pi probably runs 32bit.
The issue is mentioned in requests_toolbelt's code as well.
To fix this you need a more recent version of requests_toolbelt (I tested successfully with 0.6.0) and a small change to beam itself:

theProgressBar = ProgressBar(
    maxval = fileMPE.len,
    widgets = WIDGETS,
)


Answered By - Ionut Ticus
Answer Checked By - Gilberto Lyons (WPSolving Admin)
at