PromptHub
Back to Errors
Python Intermediate 13 views

Python UnicodeDecodeError

Python failed to decode bytes into a string using the specified encoding.

Explanation

UnicodeDecodeError occurs when Python tries to decode a bytes object into a string but encounters bytes that are not valid for the specified encoding. For example, trying to decode Latin-1 encoded bytes as UTF-8. This commonly happens when reading files with unknown encodings, processing data from different sources that use different encodings, or when binary data is incorrectly treated as text. The error message shows the exact byte position where decoding failed.

Common Causes

  • Wrong encoding specified
  • File uses different encoding than expected
  • Binary file opened in text mode
  • Mixed encoding in data source
  • Non-UTF-8 characters in file

Solution

Specify the correct encoding when opening files: open(file, encoding='latin-1'). Try different encodings to find the correct one. Use errors parameter to handle decoding errors: open(file, encoding='utf-8', errors='ignore'). Detect encoding automatically with chardet or charset-normalizer libraries. For binary files, open in binary mode ('rb'). Use latin-1 as a fallback encoding since it can decode any byte sequence.

Code Example 

# UnicodeDecodeError when reading file with wrong encoding
with open('data.txt', 'r') as f:  # defaults to UTF-8
    content = f.read()  # UnicodeDecodeError if not UTF-8!

# Fix: specify correct encoding
with open('data.txt', 'r', encoding='latin-1') as f:
    content = f.read()

# Or handle errors gracefully
with open('data.txt', 'r', encoding='utf-8', errors='replace') as f:
    content = f.read()  # replaces invalid bytes with \ufffd

# Auto-detect encoding
import chardet
with open('data.txt', 'rb') as f:
    raw = f.read()
    detected = chardet.detect(raw)
    print(detected)  # {'encoding': 'windows-1252', 'confidence': 0.73}

with open('data.txt', 'r', encoding=detected['encoding']) as f:
    content = f.read()

# For web responses
response.encoding = response.apparent_encoding

Error Information

Language

python

Difficulty

Intermediate

Views

13

Related Errors

Python

Python ValueError

A function received an argument of correct type but inappropriate value.
Python

Python KeyError

The code tries to access a dictionary key that doesn't exist.
Python

Python AttributeError

The code tries to access an attribute or method that doesn't exist on an object.
Python

Python ImportError: cannot import name

The module exists but the specific name (function, class) cannot be imported from it.