Generic.Files.LineLength miscalculates the length of line containing multibyte characters

[julien-picalausa]

Describe the bug

Generic.Files.LineLength will report a line to be over the limit if the line is at limit and contains one or more multibyte characters. This is easily reproducible when having UTF-8 strings in the code base.

Using

echo "Jeg liker blåbærsyltetøy";

Custom ruleset

<?xml version="1.0"?>
<ruleset name="My Custom Standard">
  <description>If you are using a custom ruleset, please enter it here.</description>
  <rule ref="Generic.Files.LineLength">
      <properties>
          <property name="lineLimit" value="33" />
          <property name="absoluteLineLimit" value="33" />
      </properties>
  </rule>
</ruleset>

To reproduce

Steps to reproduce the behavior:

1. Create a file called test.php with the code sample above...
2. Run phpcs test.php ...
3. See error message displayed

 1  | ERROR | [ ] Line exceeds maximum limit of 33 characters; contains 35 characters

Expected behavior

No error. The line only has 32 characters (even though it is 35 bytes long)

Versions (please complete the following information)

Operating System	FreeBSD 13.1
PHP version	PHP 8.2.12
PHP_CodeSniffer version	PHP_CodeSniffer version 3.7.2 (stable)
Standard	custom
Install type	PHAR

Additional context

None I can think og

Please confirm:

• I have searched the issue list and am not opening a duplicate issue.
• I confirm that this bug is a bug in PHP_CodeSniffer and not in one of the external standards.
• I have verified the issue still exists in the master branch of PHP_CodeSniffer.

[oraxisart1]

Facing same issue on Manjaro Linux

[jrfnl]

@julien-picalausa @oraxisart1 I just happened to come across this issue and tried to reproduce it. The only way I seem to be able to reproduce it, is if a) the file is saved as UTF-8 encoded + b) the PHP iconv extension is not available.

Can you confirm that on your systems the iconv extension is not available ?

Suggested solution: turn on the iconv extension if you want to analyse code in files which contain non-ascii characters.

[julien-picalausa]

I can confirm that I do not have iconv installed and probably didn't have it back then either. I currently don't have any environment in place to test it, unfortunately.

What I can see however is that there is no clear documentation of this iconv requirement, so it's either missing or easy to miss. If this requirement could be mentioned prominently, I believe that would be sufficient to address this issue.

[jrfnl]

What I can see however is that there is no clear documentation of this iconv requirement, so it's either missing or easy to miss. If this requirement could be mentioned prominently, I believe that would be sufficient to address this issue.

The iconv extension is not a hard requirement, but it is strongly recommended and for non-ascii based codebases, some sniffs may give incorrect results if the iconv extension is not available.
Having said that, it is only a very small subset of sniffs which is affected and most things in PHPCS will work just fine without the iconv extension.

@julien-picalausa Would you be willing to submit a PR to improve the documentation page ?
Documentation improvements can be submitted to this repo: https://github.com/PHPCSStandards/PHP_CodeSniffer-documentation

[rodrigoprimo]

I stumbled upon this issue while reviewing some issues in this repository and took the liberty to create the PR suggested above: PHPCSStandards/PHP_CodeSniffer-documentation#91