Feature request: support locales set in the system

[mowgli]

I always use latin1 encoding in all my systems and all my file systems. On the other hand, I use task for many year now up to version 2.6. However, the code did corrode a bit and something like task all refuses to work now. So I always had a look at the upcoming v3 version.

However, it does not work correctly and I am not able to get it to work with German umlauts.

A ü will be convertet to � and so on. Even more, it is not possible to use any other chars than ASCII.

Migrating all my systems to utf-8 is not a option as that would break so many other stuff and making many files impossible to use anymore.

But this is exactly why there is locales. Mine is for example:

LANG=de_DE
LC_CTYPE=de_DE
LC_NUMERIC=de_DE
LC_TIME=de_DE
LC_COLLATE=de_DE
LC_MONETARY=de_CH
LC_MESSAGES=de_DE
LC_PAPER=de_DE
LC_NAME=de_DE
LC_ADDRESS=de_CH
LC_TELEPHONE=de_DE
LC_MEASUREMENT=de_DE
LC_IDENTIFICATION=de_DE
LC_ALL=

So I want to ask to care about the user set locales and not blindly convert anything to utf8 (It is even not proper utf8 if I see correct.)

If there is any that I can do, I'd like to do. I have good knowledge in C, moderate in C++ and excelent in perl but none in python.

I tried to find where the input/output is done in the code but I got lost. In my opinion it should be pretty easy to do the conversion in input/output.

Please consider that request a try. I would love to use task further on as it is exactly filling my needs.

[djmitche]

Rust does require that strings are valid UTF-8, but AIUI Taskwarrior always uses UTF-8 when storing task data, converting from the input encoding. I wonder if this is not actually thecase: perhaps v2 was not doing any conversion, and just assuming the latin-1 was actually UTF-8, then returning it byte-for-byte. That might cause issues where the latin-1 is not valid UTF-8!

Could you provide a reproduction recipe for this?

A brief experiment shows

>>> "�".encode("utf-8")
b'\xc3\xaf\xc2\xbf\xc2\xbd'
>>> "�".encode("utf-8").decode("latin-1")
'�'
>>> "ü".encode("latin-1")
b'\xfc'
>>> "ü".encode("latin-1").decode("utf-8")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 0: invalid start byte
>>> "�".encode("latin-1").decode("utf-8")
'�'
>>> "ü".encode("utf-8").decode("latin-1")
'ü'

but these things are hard to debug without access to the actual bytes being handled.

[mowgli]

Hi, Am So den 1. Mär 2026 um 4:19 schrieb Dustin J. Mitchell:

Rust does require that strings are valid UTF-8, but AIUI Taskwarrior always uses UTF-8 when storing task data, converting from the input encoding. I wonder if this is not actually thecase: perhaps v2 was not doing any conversion, and just assuming the latin-1 was actually UTF-8, then returning it byte-for-byte. That might cause issues where the latin-1 is not valid UTF-8!

I am not sure when it comes to v2. Many bytes in latin1 can never be mapped to utf-8 (but the oposide is possible as all bytes in latin1 are defined.) So I believe that v2 just uses the byte values and never does some encoding stuff. When it comes to v3, I am fully ok with storing the stuff in utf-8. But the encoding from input and output should be the locale one.

Could you provide a reproduction recipe for this? A brief experiment shows ``` >>> "�".encode("utf-8") b'\xc3\xaf\xc2\xbf\xc2\xbd'

I get `b'\xef\xbf\xbd'`! So python does something fishy with that string.

>>> "�".encode("utf-8").decode("latin-1") '�'

I get `'�'`

>>> "ü".encode("latin-1") b'\xfc'

python is not able to work further when I type `ü` and it is not even possible to end python anymore. So I switched to perl and was able to get your results. Here is my script, be sure to save it in latin1 encoding! ``` #! /usr/bin/perl use Encode; my $string = "�"; print "'", encode('UTF-8', $string), "'\n"; print "'", encode('UTF-8', "ü"), "'\n"; print "'", decode('latin1', "ü"), "'\n"; print "'", decode('latin1', encode('UTF-8', "ü")), "'\n"; print "'", decode('UTF-8', encode('latin1', "ü")), "'\n"; ``` Please note, that the last string is exactly "�" So the encoding in the older version seems to be the same than the following.

>>> "ü".encode("latin-1").decode("utf-8")

I did some more tests with again updating to v3.4.2 and converting the old database. First I created a new database. When I tried so, I seen that I was not able to create a task with an `ß` which was possible in the past. ``` ~> task add Test of ß /usr/lib/gcc/x86_64-pc-linux-gnu/14/include/g++-v14/bits/basic_string.h:1269: std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::reference std::__cxx11::basic_string<_CharT, _Traits, _Alloc>::operator[](size_type) [with _CharT = char; _Traits = std::char_traits<char>; _Alloc = std::allocator<char>; reference = char&; size_type = long unsigned int]: Assertion '__pos <= size()' failed. ``` Maybe that explains why `task all` is not possible anymore. So, I have the following task now: ``` ID Age Urg Description 1 2min 7 Test of äöü ``` Updating to v3.4.2 and `task import-v2 rc.hooks=0` Now it shows: ``` ID Age Urg Description 1 14min 7 Test of ��� ``` I tried to edit the task replacing the string `���` with äöü in latin1 encoding doesn't change the string in task. I wonder what that should be as when converting it from UTF8 to latin1 gives `???`, so no convertion possible. When I change the editor to utf-8, I get the following: ``` ID Age Urg Description 1 16min 7 Test of äöü ``` So no way to enter latin1 or even display latin1 in v3. Regards Klaus -- Klaus Ethgen http://www.ethgen.ch/ pub 4096R/4E20AF1C 2011-05-16 Klaus Ethgen ***@***.***> Fingerprint: 85D4 CA42 952C 949B 1753 62B3 79D0 B06F 4E20 AF1C

[djmitche]

OK, with a little debugging I can confirm that the latin-1-encoded string is being passed into TaskChampion (the Rust library) without being re-encoded. That's calling CxxString::to_string_lossy which is replacing it with the unicode replacement character, U+FFFD. And that's exactly what you're seeing:

>>> "\uFFFD".encode('utf-8').decode('latin-1')
'�'

so task is outputting the UTF-8 representation of U+FFFD, and your terminal is interpreting that as latin-1. BTW, I wrote a guide to debugging encoding issues a few years ago -- http://code.v.igoro.us/posts/2022/01/encodings.html.

Looking at src/Lexer.cpp, it appears to assume that the command-line input is in UTF-8:

taskwarrior/src/Lexer.cpp

Lines 52 to 54 in 13a8d80

	bool Lexer::token(std::string& token, Lexer::Type& type) {
	// Eat white space.
	while (unicodeWhitespace(_text[_cursor])) utf8_next_char(_text, _cursor);

So in 2.x that utf-8 was handled "loosely", meaning that invalid UTF-8 like [0xfc] did not cause errors, and was stored as-is in the task DB, then reproduced as-is on output, although probably with some alignment issues in reports. 3.x expects stored data to be valid UTF-8, so this "hack" no longer works.

At the very least, we should probably detect this issue and warn about it. Can you run the following C++ program and report the results?

#include <langinfo.h>
#include <locale.h>

#include <iostream>

int main() {
  if (!std::setlocale(LC_CTYPE, "")) {
    std::cerr << "Error: Could not set locale from environment.\n";
  }

  char* encoding = nl_langinfo(CODESET);
  std::cerr << "The determined encoding is " << encoding << "\n";
}

For me:

⸩ g++ -o foo main.cpp 
⸩ ./foo 
The determined encoding is UTF-8

(this is correct -- I use en_US.utf8)

[mowgli]

Hi, Am Mo den 2. Mär 2026 um 19:18 schrieb Dustin J. Mitchell:

so `task` is outputting the UTF-8 representation of U+FFFD, and your terminal is interpreting that as latin-1. BTW, I wrote a guide to debugging encoding issues a few years ago -- http://code.v.igoro.us/posts/2022/01/encodings.html.

I will have a look at that guide... The issues get more and more.

So in 2.x that utf-8 was handled "loosely", meaning that invalid UTF-8 like `[0xfc]` did not cause errors, and was stored as-is in the task DB, then reproduced as-is on output, although probably with some alignment issues in reports. 3.x expects stored data to be valid UTF-8, so this "hack" no longer works. At the very least, we should probably detect this issue and warn about it. Can you run the following C++ program and report the results?

I agree with that. And better, handle the output properly. (For example via iconv.)

For me: ``` ??? g++ -o foo main.cpp ??? ./foo The determined encoding is UTF-8 ``` (this is correct -- I use en_US.utf8)

For me: ``` ~> ./a.out The determined encoding is ISO-8859-1 ``` which is also correct. Regards Klaus -- Klaus Ethgen http://www.ethgen.ch/ pub 4096R/4E20AF1C 2011-05-16 Klaus Ethgen ***@***.***> Fingerprint: 85D4 CA42 952C 949B 1753 62B3 79D0 B06F 4E20 AF1C

[djmitche]

I just added a topic:encodings label to gather issues related to this.

It seems like Taskwarrior assumes a UTF-8 encoding, and always has. (ref, ref) For example, it assumes strings are UTF-8 when calculating their widths. I don't think that's an unreasonable design decision, although it was made before my time! It was not unreasonable then, and is even less unreasonable now.

Although latin-1 support was already a bit broken, 3.x made things worse for folks such as yourself who are not using utf-8. This is because it doesn't pass through invalid UTF-8-encoded values (that is, the upper 128 latin-1 characters), but replaces them with U+FFFD, which when output to a latin-1 terminal mojibakes to �.

Fixing this comprehensively is a pretty substantial bit of work, and likely to lead to more bugs in the short term -- it would require decoding the locale encoding for all inputs (CLI, maybe task edit? import??) and encoding back to the locale encoding for output (terminal, producing the file for task edit, export?). There are a lot of unclear edge cases there around unusual environments like WSL and differences between OS implementations (some of the other topic:encodings bugs suggest things work differently on MacOS, for example).

I'm not sure if there's another, less-invasive way to fix the new issues which wouldn't involve more hacks. Even correctly decoding latin-1 in command-line args to Unicode isn't enough -- the umlaut would encode to C3 BC, which when interpreted as latin-1 is ü.

I'm interested what others think!