Python - OSError: [Errno 24] Too many open files

The "Too many open files" error in Python typically occurs when your program has opened more file descriptors than the operating system allows. This error is often seen in situations where files are not being properly closed after use, leading to the exhaustion of available file descriptors.

Close Files Properly

Make sure you are closing files after you finish using them. Use the close() method for file objects or, preferably, use the with statement, which automatically closes the file when you are done with it:

with open('file.txt', 'r') as file:
   # Do something with the file
# File is automatically closed when exiting the 'with' block

The script that is opening the files

def get_data_from_json(source_file):
  # initialise data
  data = {}

  with open(source_file, 'r') as json_file:
    data = json.load(json_file)

  return data

The with statement in Python automatically takes care of closing the file when the block is exited, whether it is exited normally or due to an exception.  The with statement ensures that the file is properly closed when you're done with it, even if an exception occurs inside the block. There's no need for an explicit call to close() in this context. The with statement is recommended because it provides a more concise and Pythonic way of handling resources, such as files.

Check for Resource Leaks

Review your code to identify any potential resource leaks, such as unclosed file handles. Ensure that you are not unintentionally leaving files open.

Increase the Open File Limit

If you are dealing with a large number of files, you may need to increase the open file limit. You can do this using the ulimit command in the terminal or shell. Such as increasing the limit to 4096

ulimit -n 4096

Keep in mind that this solution might be a temporary fix, and you should investigate why your code is opening so many files.

Limit File Openings

If you are processing a large number of files, consider processing them in smaller batches or optimizing your code to avoid opening too many files simultaneously.

Handle Exceptions

Implement proper exception handling in your code to ensure that files are closed even if an exception occurs.