PromptHub
Back to Errors
Python Intermediate 108 views

Python UnicodeDecodeError

Python failed to decode bytes into a string using the specified encoding.

Explanation

UnicodeDecodeError occurs when Python tries to decode a bytes object into a string but encounters bytes that are not valid for the specified encoding. For example, trying to decode Latin-1 encoded bytes as UTF-8. This commonly happens when reading files with unknown encodings, processing data from different sources that use different encodings, or when binary data is incorrectly treated as text. The error message shows the exact byte position where decoding failed.

Common Causes

  • Wrong encoding specified
  • File uses different encoding than expected
  • Binary file opened in text mode
  • Mixed encoding in data source
  • Non-UTF-8 characters in file

Solution

Specify the correct encoding when opening files: open(file, encoding='latin-1'). Try different encodings to find the correct one. Use errors parameter to handle decoding errors: open(file, encoding='utf-8', errors='ignore'). Detect encoding automatically with chardet or charset-normalizer libraries. For binary files, open in binary mode ('rb'). Use latin-1 as a fallback encoding since it can decode any byte sequence.

Code Example 

# UnicodeDecodeError when reading file with wrong encoding
with open('data.txt', 'r') as f:  # defaults to UTF-8
    content = f.read()  # UnicodeDecodeError if not UTF-8!

# Fix: specify correct encoding
with open('data.txt', 'r', encoding='latin-1') as f:
    content = f.read()

# Or handle errors gracefully
with open('data.txt', 'r', encoding='utf-8', errors='replace') as f:
    content = f.read()  # replaces invalid bytes with \ufffd

# Auto-detect encoding
import chardet
with open('data.txt', 'rb') as f:
    raw = f.read()
    detected = chardet.detect(raw)
    print(detected)  # {'encoding': 'windows-1252', 'confidence': 0.73}

with open('data.txt', 'r', encoding=detected['encoding']) as f:
    content = f.read()

# For web responses
response.encoding = response.apparent_encoding

Error Information

Language

python

Difficulty

Intermediate

Views

108

Related Errors

Python

Python EOFError

The input() function encountered end-of-file (no more data to read).
Python

Python TabError

The code uses inconsistent indentation by mixing tabs and spaces.
Python

Python AttributeError

The code tries to access an attribute or method that doesn't exist on an object.
Python

Python NameError: name is not defined

The code references a variable or function name that hasn't been defined.
All tools