PromptHub
Back to Errors
Python Intermediate 186 views

Python UnicodeDecodeError

Python failed to decode bytes into a string using the specified encoding.

Explanation

UnicodeDecodeError occurs when Python tries to decode a bytes object into a string but encounters bytes that are not valid for the specified encoding. For example, trying to decode Latin-1 encoded bytes as UTF-8. This commonly happens when reading files with unknown encodings, processing data from different sources that use different encodings, or when binary data is incorrectly treated as text. The error message shows the exact byte position where decoding failed.

Common Causes

  • Wrong encoding specified
  • File uses different encoding than expected
  • Binary file opened in text mode
  • Mixed encoding in data source
  • Non-UTF-8 characters in file

Solution

Specify the correct encoding when opening files: open(file, encoding='latin-1'). Try different encodings to find the correct one. Use errors parameter to handle decoding errors: open(file, encoding='utf-8', errors='ignore'). Detect encoding automatically with chardet or charset-normalizer libraries. For binary files, open in binary mode ('rb'). Use latin-1 as a fallback encoding since it can decode any byte sequence.

Code Example 

# UnicodeDecodeError when reading file with wrong encoding
with open('data.txt', 'r') as f:  # defaults to UTF-8
    content = f.read()  # UnicodeDecodeError if not UTF-8!

# Fix: specify correct encoding
with open('data.txt', 'r', encoding='latin-1') as f:
    content = f.read()

# Or handle errors gracefully
with open('data.txt', 'r', encoding='utf-8', errors='replace') as f:
    content = f.read()  # replaces invalid bytes with \ufffd

# Auto-detect encoding
import chardet
with open('data.txt', 'rb') as f:
    raw = f.read()
    detected = chardet.detect(raw)
    print(detected)  # {'encoding': 'windows-1252', 'confidence': 0.73}

with open('data.txt', 'r', encoding=detected['encoding']) as f:
    content = f.read()

# For web responses
response.encoding = response.apparent_encoding

Error Information

Language

python

Difficulty

Intermediate

Views

186

Related Errors

Python

Python AttributeError

The code tries to access an attribute or method that doesn't exist on an object.
Python

Python IndexError: list index out of range

The code tries to access a list index that is beyond the list's length.
Python

Python RecursionError

The function calls itself too many times, exceeding the maximum recursion depth.
Python

Python IndentationError

Python code has inconsistent or incorrect indentation.
All tools