### Japanese charsets

Setting up Japanese charsets is quite difficult. This is mainly
because:

1. Windows character set is extended from the original legacy Japanese
   standard (JIS X 0208) and is not standardized.
  This means that the strictly standardized implementation cannot
  support full Windows character set.

2. There are several encoding methods in Japanese
  Mainly for historical reasons, there are several encoding methods in
  Japanese, which are not fully compatible with each other. There are
  two major encoding methods. One is Shift_JIS series used in Windows
  and some UNIXes and the other is EUC-JP series used in most UNIXes
  and Linux. Moreover, prior to Samba 2.2 series also offers several
  unique encoding method called CAP and HEX to keep interoperability
  with CAP/NetAtalk and UNIXes which cannot use a Japanese file name. 
  Some implementation of EUC-JP series cannot support full Windows
  character set.

3. There are some code conversion tables between Unicode and legacy
   Japanese character sets

  There are some code conversion tables between Unicode and legacy
  Japanese character sets, one is compatible with Windows, another one
  is refered to the reference of Unicode consocium and others are 
  the mixed implementation... Unicode cnsocium does not officially
  defined any conversion tables between Unicode and legacy character
  sets so there cannot be standard one.

4. The implementation of iconv() depends on the system

  The character set and conversion tables implemented in iconv()
  depends on the system. Also the Japanese locale names may be
  differenct on the different systems.
  This means that the value of charset parameters depends on the
  implementation of iconv() or the system.

Though 2 byte fixed UCS-2 encoding is used in Windows internal,
Shift_JIS series encoding is usually used in Japanese environment
as ASCII encoding in English environment.

## The basic parameter setting

dos charset and display charset should be set to the locale compatible
with the character set and encoding method used on Windows. This is
usually CP932 but is sometimes a different name.

unix charset can be selected into Shift_JIS series, EUC-JP series and
UTF-8. UTF-8 is always available but the availability of other locales
and its name itself depends on the system.

Additionally it is considerable to select Shift_JIS series as the
value of unix charset parameter with using vfs_cap module like "coding
system = CAP" in Samba 2.2 series.

How to set unix charset parameter is a difficult question. IHere is
the detail and merit and demerit for each value of "unix charset".

o Shift_JIS series
  Shift_JIS series means a locale which is equivalent to "Shift_JIS", used as a
  standard on Japanese Windows. In the case of Shift_JIS, for example
  if a Japanese file name consist of 0x8ba4 and 0x974c (a 4 bytes
  Japanese character string meaning "share") and ".txt" is written
  from Windows on Samba, the file name on UNIX becomes 0x8ba4, 0x974c,
  ".txt" (a 8 bytes BINARY string), same as Windows.

  Since Shift_JIS series is usually used on some commercial based
  UNIX, hp-ux and AIX as Japanese locale (however, it is also possible
  to use EUC-JP series), To use Shift_JIS series on these platforms,
  Japanese file names created from Windows can be referred to also on
  UNIX.

  If your UNIX is already working with Shift_JIS and there is a user 
  who needs to use Japanese file names written from Windows, basically
  Shift_JIS series is the best choice.

  However, broken file names may be displayed and some commands which
  cannot handle non-ASCII filenames may be aborted during parsing
  filenames. especially there may be "\ (0x5c)" in file names, which
  need to be handled carefully. So you had better not touch file names
  written from Windows on UNIX.

  Notes that most Japanized free softwares work actually with EUC-JP
  only. You had better confirm to use if the Japanized free software
  can work with Shift_JIS.

o EUC-JP series
  EUC-JP series means a locale which is equivalent to the industry
  standard called EUC-JP, widely used in Japanese UNIX (although EUC
  contains specifications for langauages other than Japanese, such as
  EUC-KR). In the case of EUC-JP series, for example if a Japanese
  file name consist of 0x8ba4 and 0x974c and ".txt" is written from
  Windows on Samba, the file name on UNIX becomes 0xb6a6, 0xcdad,
  ".txt" (a 8 bytes BINARY string). 

  Since EUC-JP is usually used on Open source UNIX, Linux and FreeBSD,
  and on commercial based UNIX, Solaris, IRIX and Tru64 UNIX as
  Japanese locale (however, it is also possible on Solaris to use
  Shift_JIS and UTF-8, on Tru64 UNIX to use Shift_JIS). To use EUC-JP
  series, most Japanese file names created from Windows can be
  referred to also on UNIX. Also, most Japanized free softwares work
  mainly with EUC-JP only. 

  It is recommended to choose EUC-JP series when using Japanese file
  names on these UNIX.

  Although there is no character which needs to be carefully treated
  like "\ (0x5c)", broken file names may be displayed and some
  commands which cannot handle non-ASCII filenames may be aborted
  during parsing filenames.

  Moreover, if you built Samba using differently installed libiconv,
  eucJP-ms locale included in libiconv and EUC-JP series locale
  included in OS may not be compatible. In this case, you may need to
  avoid using incompatible characters for file names.

o UTF-8
  UTF-8 means a locale which is equivalent to UTF-8, the international
  standard defined by Unicode consosium. In UTF-8, a *character* is
  expressed with 1 - 3 *bytes*. In case of Japanese, most characters
  are expressed with 3 bytes. Since on Windows Shift_JIS, where a
  character is expressed with 1 or 2 bytes, is used to express
  Japanese, basically a byte length of a UTF-8 string grows 1.5 times
  the length of a original Shift_JIS string. In the case of UTF-8,
  for example if a Japanese file name consist of 0x8ba4 and 0x974c and
  ".txt" is written from Windows on Samba, the file name on UNIX
  becomes 0xe585, 0xb1e6, 0x9c89, ".txt" (a 10 bytes BINARY string).

  For systems where iconv() is not available or where iconv()'s locales
  are not compatible with Windows, UTF-8 is the only locale available.

  There is no systems which uses UTF-8 as default locale for Japanese.

  Some broken file names may be displayed and some commands which
  cannot handle non-ASCII filenames may be aborted during parsing
  filenames. especially there may be "\ (0x5c)" in file names, which
  need to be handled carefully. So you had better not touch file names
  written from Windows on UNIX.

  In addition, although it is not directly concerned with Samba, since
  there is a delicate difference between iconv() function, which is
  generally used on UNIX and the functions used on other platforms,
  such as Windows and Java about the conversion table between
  Shift_JIS and Unicode, you should be carefully to handle UTF-8.

  Although Mac OS X uses UTF-8 as its encoding method for filenames,
  it uses an extended UTF-8 specification that Samba cannot handle so
  UTF-8 locale is not available for Mac OS X.

o Shift_JIS series + vfs_cap (CAP encoding)
  CAP encoding means a specification using in CAP and NetAtalk, file
  server softwares for Macintosh. In the case of CAP encoding, for
  example if a Japanese file name consist of 0x8ba4 and 0x974c and
  ".txt" is written from Windows on Samba, the file name on UNIX
  becomes ":8b:a4:97L.txt" (a 14 bytes ASCII string). 

  For CAP encoding a byte which cannot be expressed as an ASCII
  character (0x80 or above) is encoded as ":xx" form. You need to take
  care of containing a "\(0x5c)" in a filename but filenames are not
  broken in a system which cannot handle non-ASCII filenames.

  The greatest merit of CAP encoding is the compatibility of encoding
  filenames with CAP or NetAtalk, file server softwares of Macintosh.
  Since they usually write a file name on UNIX with CAP encoding, if a
  directory is shared with both Samba and NetAtalk, you need to use
  CAP encoding to avoid non-ASCII filenames are broken.

  However, recently there are some systems where the Netatalk which is
  applied a patch to write filenames with EUC-JP is installed
  (i.e. Japanese original Vine Linux), where you need to choose EUC-JP
  series instead of CAP encoding.

  vfs_cap itself is available for non Shift_JIS series locales for
  systems which cannot handle non-ASCII characters or systems which
  shares files with NetAtalk.

  To use CAP encoding on Samba 3.0, you should use unix charset
  parameter and VFS like:

-----
[global]
  dos charset = CP932
  unix charset = CP932

  ...

[cap-share]
  vfs objects = cap

(*) the locale name "CP932" may be different name.
-----

  You should set CP932 if using GNU libiconv for unix charset. Setting
  this, filenames in the "cap-share" share are written with CAP
  encoding.

## Individual implementations

Here are some additional informations about indivisual implementations:

o GNU libiconv
  To handle Japanese correctly, you should apply a patch(*) to
  libiconv-1.8.

(*)libiconv-1.8-cp932-patch.diff.gz
  http://www2d.biglobe.ne.jp/~msyk/software/libiconv-patch.html

  Using the patched libiconv-1.8, these settings are available:

  dos charset = CP932
  unix charset = CP932 / eucJP-ms / UTF-8
                 |       |
                 |       +-- EUC-JP series
                 +-- Shift_JIS series
  display charset = CP932

  Other Japanese locales (for example Shift_JIS and EUC-JP) should not
  be used for the lack of the compatibility with Windows.

o GNU glibc
  To handle Japanese correctly, you should apply a patch(*) to
  glibc-2.2.5/2.3.1/2.3.2 or should use the patch-merged versions,
  glibc-2.3.3 or later.

(*) http://www2d.biglobe.ne.jp/~msyk/software/glibc/

  Using the above glibc, these setting are available:

  dos charset = CP932
  unix charset = CP932 / eucJP-ms / UTF-8
  display charset = CP932

  Other Japanese locales (for example Shift_JIS and EUC-JP) should not
  be used for the lack of the compatibility with Windows.

o Solaris 9

o hp-ux 11i

o AIX 5L

o Tru64 UNIX 5.1

## Migration from Samba 2.2 series

  Prior to Samba 2.2 series "coding system" parameter is used as "unix
  charset" parameter of Samba 3.0 series. The table below shows the
  mapping table when migrating from Samba 2.2 series to Samba 3.0
  series.

  Samba 2.2     Samba 3.0
  coding system unix charset  
  ------------- ------------    
  SJIS,	        Shift_JIS series
  EUC,          EUC-JP series
  EUC3(*),      EUC-JP series
  CAP,          Shift_JIS series + VFS
  HEX,          currently none 
  UTF8,         UTF-8
  UTF8-Mac(*),  currently none
  others,       none

(*) Only exists in Samba Japanese Edition