feat: add -f codepage flag for input/output encoding #638

dlevy-msft-sql · 2026-01-26T00:12:41Z

Summary

Implements the -f flag from ODBC sqlcmd for specifying input and output file code pages.

Changes

cmd/sqlcmd/sqlcmd.go: Added CodePage and ListCodePages fields, -f/--code-page flag, validation, and runtime application
pkg/sqlcmd/codepage.go: Comprehensive codepage support with parsing and encoding conversion
pkg/sqlcmd/codepage_test.go: Unit tests for codepage parsing and supported encodings
pkg/sqlcmd/commands.go: Updated :R command to respect codepage settings when reading files
pkg/sqlcmd/sqlcmd.go: Added CodePage field to Sqlcmd struct
cmd/sqlcmd/sqlcmd_test.go: Command line argument tests for valid and invalid codepage values
README.md: Documentation with format examples

Usage

# Single codepage for both input and output
sqlcmd -S server -f 65001 -i script.sql -o results.txt

# Different codepages for input and output
sqlcmd -S server -f i:1252,o:65001 -i windows_script.sql -o utf8_results.txt

# List all supported codepages
sqlcmd --list-codepages

Supported Codepages

Unicode: 65001 (UTF-8), 1200 (UTF-16LE), 1201 (UTF-16BE)
Windows: 874, 1250-1258
OEM/DOS: 437, 850, etc.
ISO-8859: 28591-28606
CJK: 932 (Shift-JIS), 936 (GB2312), 949 (Korean), 950 (Big5)
EBCDIC: 37, 1047, 1140

Testing

All existing tests pass
Added command line argument parsing tests
Added unit tests for codepage parsing and validation

Improves ODBC sqlcmd compatibility.

dlevy-msft-sql · 2026-01-26T00:13:51Z

Supersedes #627

Copilot

Pull request overview

This PR adds configurable input/output code page handling to the legacy sqlcmd CLI, improving compatibility with ODBC sqlcmd and enabling legacy-encoding workflows.

Changes:

Introduces a central code page registry and parsing logic (-f / --code-page) with support for many Windows, OEM, ISO-8859, CJK, and EBCDIC encodings and a --list-codepages listing mode.
Wires code page settings through the legacy CLI into the core Sqlcmd engine so :R, :OUT, and :ERROR commands correctly decode input files and encode output/error files according to the configured code pages.
Adds comprehensive unit tests for code page parsing/lookup and CLI argument validation, plus README documentation and examples for the new functionality.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
`pkg/sqlcmd/sqlcmd_test.go`	Adds tests to verify `IncludeFile` decodes input files correctly under various input code pages (Windows-1252, UTF-16LE/BE with and without BOM).
`pkg/sqlcmd/sqlcmd.go`	Extends `Sqlcmd` with a `CodePage` field and updates `IncludeFile` to select an appropriate `io.Reader` based on configured input code page or BOM auto-detection.
`pkg/sqlcmd/commands_test.go`	Adds tests confirming `:OUT` and `:ERROR` commands honor the configured output code page when writing to files.
`pkg/sqlcmd/commands.go`	Introduces a `transformWriteCloser` helper and updates `:OUT`/`:ERROR` commands to encode output using the selected code page while ensuring underlying files are closed correctly.
`pkg/sqlcmd/codepage_windows.go`	Implements Windows-specific `encoding.Encoding` support backed by `MultiByteToWideChar`/`WideCharToMultiByte` for system-installed code pages not in the built-in registry.
`pkg/sqlcmd/codepage_test.go`	Adds tests for `ParseCodePage`, `GetEncoding`, `SupportedCodePages`, and Windows-specific fallback behavior, including round-trip checks for a Windows-only EBCDIC code page.
`pkg/sqlcmd/codepage_other.go`	Provides a non-Windows implementation of `getSystemCodePageEncoding` that surfaces a clear error for unsupported system code pages.
`pkg/sqlcmd/codepage.go`	Defines the cross-platform code page registry, `CodePageSettings`, parsing/validation logic for `-f`, and a `SupportedCodePages` API used by the CLI.
`cmd/sqlcmd/sqlcmd_test.go`	Extends CLI argument tests to cover `-f` / `--code-page`, `--list-codepages`, and various invalid code page argument combinations.
`cmd/sqlcmd/sqlcmd.go`	Adds `CodePage` and `ListCodePages` arguments, validates and stores parsed `CodePageSettings`, applies them to the `Sqlcmd` instance, and implements `--list-codepages` output formatting.
`README.md`	Documents the new `-f` flag, describes supported code pages, provides usage examples, and explains `--list-codepages` and default BOM auto-detection behavior.

pkg/sqlcmd/codepage_windows.go

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

pkg/sqlcmd/sqlcmd.go

README.md

pkg/sqlcmd/codepage_windows.go

pkg/sqlcmd/codepage_test.go

Copilot

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

README.md

shueybubbles · 2026-01-28T15:06:57Z

cmd/sqlcmd/sqlcmd.go

 			err = localizer.Errorf("The -J parameter requires encryption to be enabled (-N true, -N mandatory, or -N strict).")
+		case a.CodePage != "":
+			if codePageSettings, parseErr := sqlcmd.ParseCodePage(a.CodePage); parseErr != nil {
+				err = localizer.Errorf(`'-f %s': %v`, a.CodePage, parseErr)


lizer.Errorf

there's no localizable content in the format string so don't need to use localizer

shueybubbles · 2026-01-28T15:07:34Z

cmd/sqlcmd/sqlcmd.go

+			if args.ListCodePages {
+				fmt.Println(localizer.Sprintf("Supported Code Pages:"))
+				fmt.Println()
+				fmt.Printf("%-8s %-20s %s\n", "Code", "Name", "Description")


", "Code", "Name", "Description")

does this string need to be localizable?

shueybubbles · 2026-01-28T15:08:14Z

cmd/sqlcmd/sqlcmd.go

+				os.Exit(0)
+			}
 			if len(argss) > 0 {
 				fmt.Printf("%s'%s': Unknown command. Enter '--help' for command help.", sqlcmdErrorPrefix, argss[0])


%s'%s': Unknown command. Enter '--help' for command help

does this string need to be localizable?

shueybubbles · 2026-01-28T15:10:46Z

  		err = localizer.Errorf("The -L parameter can not be used in combination with other parameters.")

should list code pages get the same exclusivity treatment as list servers?

Refers to: cmd/sqlcmd/sqlcmd.go:150 in a672f7f. [](commit_id = a672f7f, deletion_comment = False)

shueybubbles

dlevy-msft-sql requested a review from Copilot January 26, 2026 00:12

Copilot started reviewing on behalf of dlevy-msft-sql January 26, 2026 00:13 View session

dlevy-msft-sql mentioned this pull request Jan 26, 2026

Add -f codepage flag for input/output encoding #627

Closed

dlevy-msft-sql self-assigned this Jan 26, 2026

dlevy-msft-sql added sqlcmd switch switch in existing sqlcmd Size: S Small issue (less than one week effort) labels Jan 26, 2026

Copilot AI reviewed Jan 26, 2026

View reviewed changes

pkg/sqlcmd/codepage_windows.go Show resolved Hide resolved

dlevy-msft-sql requested a review from Copilot January 26, 2026 00:41

Copilot started reviewing on behalf of dlevy-msft-sql January 26, 2026 00:42 View session

dlevy-msft-sql added this to the ODBC SQLCMD Interchangeability milestone Jan 26, 2026

Copilot AI reviewed Jan 26, 2026

View reviewed changes

pkg/sqlcmd/sqlcmd.go Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

pkg/sqlcmd/codepage_windows.go Outdated Show resolved Hide resolved

pkg/sqlcmd/codepage_test.go Outdated Show resolved Hide resolved

dlevy-msft-sql requested a review from Copilot January 26, 2026 00:52

Copilot started reviewing on behalf of dlevy-msft-sql January 26, 2026 00:53 View session

Copilot AI reviewed Jan 26, 2026

View reviewed changes

dlevy-msft-sql requested a review from shueybubbles January 28, 2026 02:47

shueybubbles reviewed Jan 28, 2026

View reviewed changes

README.md Show resolved Hide resolved

shueybubbles reviewed Jan 28, 2026

View reviewed changes

shueybubbles approved these changes Jan 28, 2026

View reviewed changes

dlevy-msft-sql mentioned this pull request Jan 28, 2026

Investigate ODBC compatibility mode via environment variable #660

Open

6 tasks

dlevy-msft-sql added the in-review label Feb 2, 2026

dlevy-msft-sql force-pushed the round-out-part2 branch 2 times, most recently from 17fc3b5 to 623f029 Compare February 5, 2026 20:00

dlevy-msft-sql changed the title ~~Add -f codepage flag for input/output encoding~~ feat: add -f codepage flag for input/output encoding Feb 5, 2026

dlevy-msft-sql force-pushed the round-out-part2 branch from 65939c4 to 41133ed Compare February 5, 2026 21:57

dlevy-msft-sql removed the in-review label Feb 6, 2026

dlevy-msft-sql force-pushed the round-out-part2 branch from 41133ed to 0a65f11 Compare February 6, 2026 00:36

feat: add -f codepage flag for input/output encoding

cf7da77

dlevy-msft-sql force-pushed the round-out-part2 branch from 0a65f11 to cf7da77 Compare February 6, 2026 00:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add -f codepage flag for input/output encoding #638

feat: add -f codepage flag for input/output encoding #638

Uh oh!

dlevy-msft-sql commented Jan 26, 2026

Uh oh!

dlevy-msft-sql commented Jan 26, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

shueybubbles Jan 28, 2026

Uh oh!

shueybubbles Jan 28, 2026

Uh oh!

shueybubbles Jan 28, 2026

Uh oh!

shueybubbles commented Jan 28, 2026

Uh oh!

shueybubbles left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

feat: add -f codepage flag for input/output encoding #638

Are you sure you want to change the base?

feat: add -f codepage flag for input/output encoding #638

Uh oh!

Conversation

dlevy-msft-sql commented Jan 26, 2026

Summary

Changes

Usage

Supported Codepages

Testing

Uh oh!

dlevy-msft-sql commented Jan 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

shueybubbles Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

shueybubbles Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

shueybubbles Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

shueybubbles commented Jan 28, 2026

Uh oh!

shueybubbles left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants