1. Home
  2. Blog
  3. Software Development

Software Development

UTF-8 Encoding Explained: How Text Becomes Bytes and How to Fix Garbled Text

UTF-8 encoding stores every Unicode character as 1 to 4 bytes, and it is the standard for the web, JSON and most code. Learn how it works byte by byte, why text turns into gibberish like café, and the 6 fixes that stop encoding bugs for good.

9 min read
UTF-8 encoding explained

Key takeaways

  • UTF-8 encoding turns Unicode code points into bytes: 1 byte for ASCII, 2 for é, 3 for € and Hindi letters, 4 for most emoji.
  • Garbled text such as café means UTF-8 bytes were decoded with the wrong encoding; the data is usually still fine.
  • Even our own demo crashed with UnicodeEncodeError on a Windows console that used cp1252.
  • Always name the encoding when reading and writing files: open(path, encoding="utf-8").

Sooner or later every developer sees it: a customer named “José” appears as “José”, a rupee sign becomes “₹”, or a script dies with UnicodeDecodeError. These bugs feel random but follow simple rules. Once you understand UTF-8 encoding, you can predict and fix them in minutes. Every byte value in this guide comes from code we ran on Python 3.13.6 on Windows.

Unicode vs UTF-8: two different things

Unicode is a giant numbered list of characters. Every character has a code point, written like U+00E9 for é. UTF-8 is one way to store those numbers as bytes in files, network messages and memory. Other encodings exist, such as UTF-16 and the older Windows-1252 (cp1252), but UTF-8 dominates the web, JSON, HTML, Linux and most programming tools.

Think of Unicode as the alphabet and UTF-8 as the handwriting. The same letter can be written in different scripts, and confusion starts when the reader assumes the wrong one.

How UTF-8 encodes characters

for ch in ["A", "é", "€", "ह", "😀"]:
    b = ch.encode("utf-8")
    print(ch, f"U+{ord(ch):04X}", len(b), b.hex(" ").upper())
Character Code point UTF-8 bytes Hex UTF-16 bytes
A U+0041 1 41 2
é U+00E9 2 C3 A9 2
€ U+20AC 3 E2 82 AC 2
ह (Hindi) U+0939 3 E0 A4 B9 2
😀 U+1F600 4 F0 9F 98 80 4

The design is clever. Plain ASCII keeps its single byte, so old English text is already valid UTF-8. Bigger code points use more bytes, and the first bits of each byte say how many bytes belong to the character, so a reader always knows where one character ends. That self-synchronising design is why UTF-8 won.

Characters are not bytes

len("café")            # 4 characters
len("café".encode())   # 5 bytes

Byte counts depend on the script. We measured three short sentences:

Text Characters UTF-8 bytes UTF-16 bytes
English 28 28 56
Hindi 32 84 64
Japanese 17 51 34

UTF-8 is compact for English and code, and larger for many Asian scripts. This matters whenever a limit is set in bytes: database columns, SMS, HTTP headers, API payloads. A 100-byte limit holds 100 English letters but only around 33 Hindi ones. The same “one character is not one unit” lesson applies to AI, as our LLM tokens guide shows. Even len("👍🏽") returned 2 in Python, because a skin-tone emoji is two code points.

Why text gets garbled (mojibake)

Garbled text has a name: mojibake, from the Japanese for “character transformation”. It happens when bytes written as UTF-8 are read as something else:

"café".encode("utf-8").decode("cp1252")    # 'café'

The two bytes for é, C3 A9, mean “Ô and “©” in Windows-1252, so one character became two. If the wrong text is saved and misread again, it gets worse: our double round trip produced café. We wrote ₹500 café to a file in UTF-8 and read it back as cp1252; it came out as ₹500 café.

The good news: the original bytes are usually intact, so the damage can be reversed by repeating the mistake backwards:

"café".encode("cp1252").decode("utf-8")   # 'café'

Tip: when you see Ã, †or  patterns, think “UTF-8 read as cp1252 or Latin-1”. When you see � (the replacement character), bytes were invalid and some information may be lost.

UnicodeDecodeError explained

The opposite mistake is reading cp1252 or Latin-1 bytes as UTF-8. In Latin-1, é is a single byte E9, which is not a valid UTF-8 sequence on its own:

b"caf\xe9".decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 3: unexpected end of data

Options, from best to worst: find out the real encoding and decode with it (b"caf\xe9".decode("latin-1") returned “café”), or, if you must keep going, use errors="replace", which gave caf� and loses the character. Never silence errors on data you need to keep. Reading files correctly is covered in our Python file handling guide.

Our console surprise

While writing this article, our first demo script crashed before printing the Hindi row:

UnicodeEncodeError: 'charmap' codec can't encode character 'ह' in position 1: character maps to <undefined>

The code was correct. The problem was the output: Python reported sys.stdout.encoding as cp1252 in that Windows shell, and cp1252 has no Hindi letters. Running with the environment variable PYTHONIOENCODING=utf-8 fixed it, and Python’s UTF-8 mode (python -X utf8) is another option. It is a perfect example of how encoding bugs live at the boundaries: files, consoles, databases and networks, not the strings inside your program.

The BOM: invisible bytes at the start

Some Windows tools add a byte order mark, EF BB BF, at the start of UTF-8 files. Read with plain utf-8, it shows up as an invisible character:

"hi".encode("utf-8-sig")                     # b'\xef\xbb\xbfhi'
b"\xef\xbb\xbfhi".decode("utf-8")            # 'hi'
b"\xef\xbb\xbfhi".decode("utf-8-sig")        # 'hi'

That stray  breaks CSV headers (“name” no longer equals “name”) and JSON parsers. Read files that may have a BOM with encoding="utf-8-sig", which strips it if present. Our Python JSON guide covers loading data files safely.

Same text, different bytes: normalisation

a = "é"           # one code point U+00E9
b = "é"     # e + combining accent
print(a == b, len(a), len(b))                    # False 1 2
print(unicodedata.normalize("NFC", b) == a)      # True

Both look identical on screen, but they are different code points, so comparisons, searches and unique constraints treat them as different. Text pasted from macOS or some keyboards often uses the decomposed form. Normalise user input to NFC before comparing or storing it.

6 fixes that prevent encoding bugs

  1. Name the encoding everywhere: open(p, encoding="utf-8"). Our Windows machine reported cp1252 as its default, so relying on defaults breaks across machines.
  2. Declare UTF-8 on the web: <meta charset="utf-8"> in HTML and Content-Type: text/html; charset=utf-8 in responses. JSON is UTF-8 by specification, which our REST API best practices guide builds on.
  3. Use full Unicode in databases: in MySQL choose utf8mb4, not the older utf8, which cannot store 4-byte characters such as emoji (we did not run MySQL for this article).
  4. Decode at the edges, work in text inside: convert bytes to strings when data arrives and back when it leaves.
  5. Handle BOMs and normalise input: use utf-8-sig for unknown files and NFC for user text.
  6. Test with real non-English data: include é, ₹, Hindi, Japanese and an emoji in your test data, and check APIs, CSV exports and emails end to end.

Symptom to fix: quick lookup

You see Likely cause Fix
café, ₹, à patterns UTF-8 bytes read as cp1252 or Latin-1 Read as UTF-8; or .encode("cp1252").decode("utf-8") to repair
caf� with replacement marks Invalid bytes replaced on decode Find the source encoding; replaced characters cannot be recovered
UnicodeDecodeError Non-UTF-8 bytes read as UTF-8 Decode with the real encoding
UnicodeEncodeError when printing Console or file uses a limited encoding PYTHONIOENCODING=utf-8 or python -X utf8
First CSV column name will not match Hidden BOM () Read with utf-8-sig
Identical-looking strings are not equal Different Unicode normalisation unicodedata.normalize("NFC", s)
Emoji saved as ???? in MySQL Column uses 3-byte utf8 Switch to utf8mb4

We confirmed the first six rows on Python 3.13; the MySQL row is standard advice we did not test here. When an encoding bug crosses an API boundary, also check the Content-Type header, which should name charset=utf-8 for text responses.

Final thoughts

UTF-8 encoding is simple at heart: Unicode numbers the characters, UTF-8 stores the numbers as 1 to 4 bytes, and every garbled string is a mismatch between how bytes were written and how they were read. Name the encoding at every boundary, prefer UTF-8 end to end, and test with real multilingual text. You will rarely see café again, and when you do, you will know exactly how to reverse it.

Frequently asked questions

What is UTF-8 encoding?

UTF-8 is a variable-length encoding that stores each Unicode character as 1 to 4 bytes. ASCII characters use 1 byte, so plain English text is identical in ASCII and UTF-8.

What is the difference between Unicode and UTF-8?

Unicode assigns a number, called a code point, to every character. UTF-8 is a way of storing those numbers as bytes. UTF-16 and UTF-32 are other ways.

Why does my text show strange characters like é?

UTF-8 bytes were decoded as Windows-1252 or Latin-1. Read the data as UTF-8, or reverse the damage by encoding the garbled text back with the wrong encoding and decoding it as UTF-8.

How do I fix UnicodeDecodeError in Python?

Open the file with the encoding it was actually written in, such as encoding=latin-1 or utf-8-sig, instead of relying on the default. Use errors=replace only when losing characters is acceptable.

How many bytes does a UTF-8 character use?

One to four. ASCII uses 1 byte, most European accented letters use 2, Hindi, Chinese and many symbols use 3, and most emoji use 4.

Keep learning

Work with Stack Quora

Need help with testing, WordPress or SEO?

We help teams ship better software and grow their sites. Tell us what you are working on and we will suggest a clear next step.

Keep reading

More guides on testing, AI & modern development

Enjoyed this article? Pick a topic and jump straight into our newest tutorials.

Artificial Intelligence

BM25 Explained: How Keyword Search Ranks Results, With Python Code and Hybrid Search

BM25 explained with a pure Python implementation we ran: the formula, IDF, term saturation, the k1 and b parameters, why…

Read article →
Software Development

HTTP Methods Explained: GET, POST, PUT, PATCH, DELETE, Tested on a Live Server

HTTP methods explained with a small server we ran: GET, POST, PUT, PATCH, DELETE, HEAD and OPTIONS, safe vs idempotent,…

Read article →
DevOps

Git Bisect Explained: Find the Commit That Broke Your Code, Manually or Automatically

Git bisect explained with a tested repository: find the one bad commit among 41 in 5 test runs, automate it…

Read article →
Test Automation

pytest Markers Explained: Custom Marks, skip, skipif, xfail and -m Filters

pytest markers explained with a tested suite: register custom marks, run only smoke tests with -m, combine filters with and,…

Read article →
Software Testing

Positive vs Negative Testing: Differences, 24 Real Test Cases and a Checklist

Positive vs negative testing explained with a real signup form: 6 positive and 18 negative test cases run in pytest,…

Read article →
Databases & SQL

SQL ORDER BY Multiple Columns: ASC, DESC, NULLS LAST, Case and Custom Order

SQL ORDER BY with multiple columns explained with tested queries: mixed ASC and DESC, tie-breakers, NULLS FIRST and LAST, case-insensitive…

Read article →
Databases & SQL

SQL Running Total: SUM OVER, Per-Group Totals, Moving Sums and the RANGE Trap

How to calculate a SQL running total with SUM() OVER: per-group totals, daily cumulative sums, moving sums, percent of total,…

Read article →
Software Development

JavaScript Sort Array: Numbers, Objects, Strings and Why 10 Comes Before 5

How to sort an array in JavaScript correctly: numbers with a compare function, objects by one or more fields, strings…

Read article →
Software Development

Python String Methods: Cheat Sheet, Real Examples and the strip() Trap

Python string methods with tested examples: split, join, strip, replace, find, startswith, is-methods, casefold, partition and translate, plus the strip()…

Read article →
Software Development

Python zip() Explained: Loop Two Lists at Once, Unzip, strict=True and Traps

Python zip() pairs items from several lists in one loop. Tested examples for dicts, unzipping with *, strict=True, zip_longest, sliding…

Read article →
Artificial Intelligence

Cosine Similarity Explained: Formula, Worked Example and Python Code

Cosine similarity explained simply: the formula, a worked example by hand, pure Python and NumPy code, cosine vs Euclidean distance…

Read article →
Software Development

UTF-8 Encoding Explained: How Text Becomes Bytes and How to Fix Garbled Text

UTF-8 encoding explained with byte-level examples we ran in Python: how characters become 1 to 4 bytes, why café turns…

Read article →
DevOps

Git Squash Commits: 4 Easy Ways, Tested Step by Step (rebase -i, reset, merge)

Git squash commits explained with 4 tested methods: interactive rebase, reset –soft, merge –squash and fixup with autosquash. Before-and-after logs,…

Read article →
Test Automation

pytest parametrize Explained: 7 Patterns with Tested Examples

pytest parametrize explained with 7 tested patterns: basic cases, readable ids, stacked decorators, pytest.param with xfail and skip, indirect fixtures…

Read article →
Software Testing

Retesting vs Regression Testing: 8 Key Differences with a Real Bug Fix Example

Retesting vs regression testing explained with a real bug fix run in pytest: retesting confirmed the fix, the regression run…

Read article →
Databases & SQL

Delete Duplicate Rows in SQL: 5 Safe Methods, Tested Step by Step

How to delete duplicate rows in SQL safely: find them, preview them, then remove them with ROW_NUMBER, MIN(id), EXISTS, rowid…

Read article →
Databases & SQL

SQL RANK vs DENSE_RANK vs ROW_NUMBER: Differences with Tested Examples

SQL RANK vs DENSE_RANK vs ROW_NUMBER in one table: how each handles ties, top-N per group, the second-highest salary trap…

Read article →
Software Development

JavaScript Optional Chaining (?.) Explained: 8 Examples, ?? Defaults and Traps

JavaScript optional chaining stops “Cannot read properties of undefined” errors. 8 tested examples for properties, arrays and methods, ?? vs…

Read article →
Artificial Intelligence

RAG Chunking Strategies Explained: Fixed, Overlap, Sentence, Recursive and Headings, Measured

RAG chunking strategies compared on one document with tiktoken 0.14: fixed-size, overlap, sentence packing, recursive splitting and heading-based chunks, measuring…

Read article →
DevOps

Git Stash Explained: Save, Apply, Pop and Untracked Files, Tested

Git stash explained with commands we ran on Git 2.56: push with a message, untracked files, list and show, apply…

Read article →
Software Testing

Pairwise Testing Explained: How All-Pairs Cuts 216 Tests to 13, Tested

Pairwise testing explained with a generator we ran in Python 3.13: a checkout with 216 combinations covered by 13 tests,…

Read article →
Test Automation

Test Doubles Explained: Dummy, Stub, Spy, Mock and Fake, with Tested Python Code

Test doubles explained with one Python example run in pytest 9.2: dummy, stub, spy, mock and fake, when to use…

Read article →
Databases & SQL

SQL COALESCE Explained: Defaults, NULLIF, IFNULL vs ISNULL and Traps, Tested

SQL COALESCE explained with queries we ran in SQLite 3.50: first non-null values, empty strings, NULL in maths, NULLIF for…

Read article →
Databases & SQL

SQL Self Join Explained: Employee-Manager, Pairs and Row Comparisons, Tested

SQL self join explained with queries we ran in SQLite 3.50: employee and manager, earning more than the manager, unique…

Read article →
Artificial Intelligence

LLM Context Window Explained: Tokens, Limits and How to Fit More, Measured

The LLM context window explained with token counts we measured using tiktoken: words vs tokens for prose, code and Hindi,…

Read article →
DevOps

Git Cherry-Pick Explained: Copy the Commit You Need, Safely, Tested

Git cherry-pick explained with commands we ran in Git 2.56: copying a hotfix to a release branch, new hashes and…

Read article →
Test Automation

Snapshot Testing Explained: How It Works, When It Helps and When It Hurts

Snapshot testing explained with a pytest and syrupy demo we ran: creating snapshots, reading the diff after a code change,…

Read article →
Software Testing

Mutation Testing Explained: Why 100% Coverage Still Missed Our Bug

Mutation testing explained with a small mutation tool we built and ran: 14 mutants, a test suite with 100% line…

Read article →
Artificial Intelligence

Prompt Injection Explained: How AI Apps Get Tricked and How to Defend Them

Prompt injection explained for builders: direct and indirect attacks, why keyword filters fail (our test blocked 1 of 4), and…

Read article →
DevOps

DNS Records Explained: A, AAAA, CNAME, MX, TXT and TTL With Real Lookups

DNS records explained with lookups we ran: how resolution works, A and AAAA, CNAME rules, MX and null MX, TXT…

Read article →
DevOps

Feature Flags Explained: Safe Releases, Gradual Rollouts and Flag Hygiene

Feature flags explained with a rollout simulation we ran: flag types, percentage rollouts with stable hashing, kill switches, allow lists,…

Read article →
Test Automation

Test Data Generation With Python Faker: Realistic, Repeatable and Safe Data

Test data generation with Python Faker, run and measured: seeding for repeatable data, locales, unique values, domain-shaped records, edge cases…

Read article →
Software Testing

Shift-Left Testing Explained: Find Bugs Earlier Without Slowing the Team

Shift-left testing explained: what it means, why early defects are cheaper, practices from requirement reviews to CI checks, the tester's…

Read article →
Artificial Intelligence

LLM Structured Output Explained: Getting Reliable JSON From AI Models

LLM structured output explained with validation we ran in Pydantic: JSON mode and schemas, constrained decoding, the ways model output…

Read article →
Test Automation

API Contract Testing Explained: Catch Breaking Changes Before They Ship

API contract testing explained with a schema check we ran: what a contract is, consumer-driven contracts, Pact, OpenAPI, which changes…

Read article →
Software Testing

Risk-Based Testing Explained: Score Risks, Build a Matrix and Test What Matters

Risk-based testing explained step by step: identifying risks, scoring likelihood and impact, building a risk matrix, mapping risk to test…

Read article →