FM / pod

/usr/share/perl5/pod

Name	Size	Mode	Actions
a2p.pod	6107	0644	edit dl rm
perl.pod	15748	0644	edit dl rm
perl56delta.pod	107184	0644	edit dl rm
perl58delta.pod	114908	0644	edit dl rm
perl561delta.pod	124697	0644	edit dl rm
perl570delta.pod	21655	0644	edit dl rm
perl571delta.pod	30382	0644	edit dl rm
perl572delta.pod	25551	0644	edit dl rm
perl573delta.pod	4644	0644	edit dl rm
perl581delta.pod	38060	0644	edit dl rm
perl582delta.pod	4470	0644	edit dl rm
perl583delta.pod	6335	0644	edit dl rm
perl584delta.pod	7363	0644	edit dl rm
perl585delta.pod	5889	0644	edit dl rm
perl586delta.pod	4651	0644	edit dl rm
perl587delta.pod	8357	0644	edit dl rm
perl588delta.pod	25267	0644	edit dl rm
perl589delta.pod	53905	0644	edit dl rm
perl590delta.pod	34951	0644	edit dl rm
perl591delta.pod	11173	0644	edit dl rm
perl592delta.pod	10954	0644	edit dl rm
perl593delta.pod	17034	0644	edit dl rm
perl594delta.pod	12661	0644	edit dl rm
perl595delta.pod	18934	0644	edit dl rm
perl5004delta.pod	56240	0644	edit dl rm
perl5005delta.pod	34285	0644	edit dl rm
perl5100delta.pod	53922	0644	edit dl rm
perl5101delta.pod	43877	0644	edit dl rm
perlaix.pod	18128	0644	edit dl rm
perlamiga.pod	7031	0644	edit dl rm
perlapi.pod	169428	0644	edit dl rm
perlapio.pod	19332	0644	edit dl rm
perlapollo.pod	829	0644	edit dl rm
perlartistic.pod	6802	0644	edit dl rm
perlbeos.pod	2939	0644	edit dl rm
perlbook.pod	663	0644	edit dl rm
perlboot.pod	28355	0644	edit dl rm
perlbot.pod	11675	0644	edit dl rm
perlbs2000.pod	7929	0644	edit dl rm
perlcall.pod	55173	0644	edit dl rm
perlce.pod	9257	0644	edit dl rm
perlcheat.pod	4153	0644	edit dl rm
perlclib.pod	7684	0644	edit dl rm
perlcn.pod	4915	0644	edit dl rm
perlcommunity.pod	6398	0644	edit dl rm
perlcompile.pod	9516	0644	edit dl rm
perlcygwin.pod	28183	0644	edit dl rm
perldata.pod	36879	0644	edit dl rm
perldbmfilter.pod	4987	0644	edit dl rm
perldebguts.pod	31887	0644	edit dl rm
perldebtut.pod	21304	0644	edit dl rm
perldebug.pod	37535	0644	edit dl rm
perldelta.pod	43877	0644	edit dl rm
perldgux.pod	2821	0644	edit dl rm
perldiag.pod	183151	0644	edit dl rm
perldoc.pod	7310	0644	edit dl rm
perldos.pod	10853	0644	edit dl rm
perldsc.pod	25447	0644	edit dl rm
perlebcdic.pod	66607	0644	edit dl rm
perlembed.pod	38033	0644	edit dl rm
perlepoc.pod	3779	0644	edit dl rm
perlfaq.pod	24841	0644	edit dl rm
perlfaq1.pod	17762	0644	edit dl rm
perlfaq2.pod	21614	0644	edit dl rm
perlfaq3.pod	38534	0644	edit dl rm
perlfaq4.pod	81412	0644	edit dl rm
perlfaq5.pod	49201	0644	edit dl rm
perlfaq6.pod	38538	0644	edit dl rm
perlfaq7.pod	37588	0644	edit dl rm
perlfaq8.pod	46128	0644	edit dl rm
perlfaq9.pod	24191	0644	edit dl rm
perlfilter.pod	21140	0644	edit dl rm
perlfork.pod	11440	0644	edit dl rm
perlform.pod	16894	0644	edit dl rm
perlfreebsd.pod	1950	0644	edit dl rm
perlfunc.pod	291909	0644	edit dl rm
perlglossary.pod	112139	0644	edit dl rm
perlgpl.pod	18806	0644	edit dl rm
perlguts.pod	105182	0644	edit dl rm
perlhack.pod	121512	0644	edit dl rm
perlhaiku.pod	1504	0644	edit dl rm
perlhist.pod	38766	0644	edit dl rm
perlhpux.pod	28650	0644	edit dl rm
perlhurd.pod	1990	0644	edit dl rm
perlintern.pod	26408	0644	edit dl rm
perlintro.pod	21491	0644	edit dl rm
perliol.pod	33755	0644	edit dl rm
perlipc.pod	67761	0644	edit dl rm
perlirix.pod	4397	0644	edit dl rm
perljp.pod	8070	0644	edit dl rm
perlko.pod	7892	0644	edit dl rm
perllexwarn.pod	14435	0644	edit dl rm
perllinux.pod	1492	0644	edit dl rm
perllocale.pod	41616	0644	edit dl rm
perllol.pod	8252	0644	edit dl rm
perlmachten.pod	4500	0644	edit dl rm
perlmacos.pod	2111	0644	edit dl rm
perlmacosx.pod	11322	0644	edit dl rm
perlmint.pod	9539	0644	edit dl rm
perlmod.pod	24494	0644	edit dl rm
perlmodinstall.pod	13929	0644	edit dl rm
perlmodlib.pod	77333	0644	edit dl rm
perlmodstyle.pod	21114	0644	edit dl rm
perlmpeix.pod	15087	0644	edit dl rm
perlmroapi.pod	2959	0644	edit dl rm
perlnetware.pod	6488	0644	edit dl rm
perlnewmod.pod	11214	0644	edit dl rm
perlnumber.pod	8352	0644	edit dl rm
perlobj.pod	21813	0644	edit dl rm
perlop.pod	96225	0644	edit dl rm
perlopenbsd.pod	1207	0644	edit dl rm
perlopentut.pod	37997	0644	edit dl rm
perlos2.pod	92783	0644	edit dl rm
perlos390.pod	16074	0644	edit dl rm
perlos400.pod	4618	0644	edit dl rm
perlothrtut.pod	40657	0644	edit dl rm
perlpacktut.pod	51047	0644	edit dl rm
perlperf.pod	51328	0644	edit dl rm
perlplan9.pod	5125	0644	edit dl rm
perlpod.pod	21626	0644	edit dl rm
perlpodspec.pod	67792	0644	edit dl rm
perlport.pod	86302	0644	edit dl rm
perlpragma.pod	4320	0644	edit dl rm
perlqnx.pod	4246	0644	edit dl rm
perlre.pod	82798	0644	edit dl rm
perlreapi.pod	25427	0644	edit dl rm
perlrebackslash.pod	20109	0644	edit dl rm
perlrecharclass.pod	21909	0644	edit dl rm
perlref.pod	26465	0644	edit dl rm
perlreftut.pod	18670	0644	edit dl rm
perlreguts.pod	36877	0644	edit dl rm
perlrepository.pod	23453	0644	edit dl rm
perlrequick.pod	17617	0644	edit dl rm
perlreref.pod	11996	0644	edit dl rm
perlretut.pod	115005	0644	edit dl rm
perlriscos.pod	1511	0644	edit dl rm
perlrun.pod	49928	0644	edit dl rm
perlsec.pod	23542	0644	edit dl rm
perlsolaris.pod	29131	0644	edit dl rm
perlstyle.pod	8618	0644	edit dl rm
perlsub.pod	54222	0644	edit dl rm
perlsymbian.pod	16222	0644	edit dl rm
perlsyn.pod	31090	0644	edit dl rm
perlthrtut.pod	46510	0644	edit dl rm
perltie.pod	36665	0644	edit dl rm
perltoc.pod	642987	0644	edit dl rm
perltodo.pod	48360	0644	edit dl rm
perltooc.pod	51425	0644	edit dl rm
perltoot.pod	68625	0644	edit dl rm
perltrap.pod	41151	0644	edit dl rm
perltru64.pod	7736	0644	edit dl rm
perltw.pod	5384	0644	edit dl rm
perlunicode.pod	56169	0644	edit dl rm
perlunifaq.pod	12878	0644	edit dl rm
perluniintro.pod	32973	0644	edit dl rm
perlunitut.pod	7907	0644	edit dl rm
perlutil.pod	9881	0644	edit dl rm
perluts.pod	3180	0644	edit dl rm
perlvar.pod	59133	0644	edit dl rm
perlvmesa.pod	3952	0644	edit dl rm
perlvms.pod	52545	0644	edit dl rm
perlvos.pod	5567	0644	edit dl rm
perlwin32.pod	40809	0644	edit dl rm
perlxs.pod	74194	0644	edit dl rm
perlxstut.pod	49670	0644	edit dl rm

Edit: /usr/share/perl5/pod/perlunitut.pod (7907B)

=head1 NAME

perlunitut - Perl Unicode Tutorial

=head1 DESCRIPTION

The days of just flinging strings around are over. It's well established that
modern programs need to be capable of communicating funny accented letters, and
things like euro symbols. This means that programmers need new habits. It's
easy to program Unicode capable software, but it does require discipline to do
it right.

There's a lot to know about character sets, and text encodings. It's probably
best to spend a full day learning all this, but the basics can be learned in
minutes.

These are not the very basics, though. It is assumed that you already
know the difference between bytes and characters, and realise (and accept!)
that there are many different character sets and encodings, and that your
program has to be explicit about them. Recommended reading is "The Absolute
Minimum Every Software Developer Absolutely, Positively Must Know About Unicode
and Character Sets (No Excuses!)" by Joel Spolsky, at
L<http://joelonsoftware.com/articles/Unicode.html>.

This tutorial speaks in rather absolute terms, and provides only a limited view
of the wealth of character string related features that Perl has to offer. For
most projects, this information will probably suffice.

=head2 Definitions

It's important to set a few things straight first. This is the most important
part of this tutorial. This view may conflict with other information that you
may have found on the web, but that's mostly because many sources are wrong.

You may have to re-read this entire section a few times...

=head3 Unicode

B<Unicode> is a character set with room for lots of characters. The ordinal
value of a character is called a B<code point>.

There are many, many code points, but computers work with bytes, and a byte can
have only 256 values. Unicode has many more characters, so you need a method
to make these accessible.

Unicode is encoded using several competing encodings, of which UTF-8 is the
most used. In a Unicode encoding, multiple subsequent bytes can be used to
store a single code point, or simply: character.

=head3 UTF-8

B<UTF-8> is a Unicode encoding. Many people think that Unicode and UTF-8 are
the same thing, but they're not. There are more Unicode encodings, but much of
the world has standardized on UTF-8.

UTF-8 treats the first 128 codepoints, 0..127, the same as ASCII. They take
only one byte per character. All other characters are encoded as two or more
(up to six) bytes using a complex scheme. Fortunately, Perl handles this for
us, so we don't have to worry about this.

=head3 Text strings (character strings)

B<Text strings>, or B<character strings> are made of characters. Bytes are
irrelevant here, and so are encodings. Each character is just that: the
character.

Text strings are also called B<Unicode strings>, because in Perl, every text
string is a Unicode string.

On a text string, you would do things like:

$text =~ s/foo/bar/;
    if ($string =~ /^\d+$/) { ... }
    $text = ucfirst $text;
    my $character_count = length $text;

The value of a character (C<ord>, C<chr>) is the corresponding Unicode code
point.

=head3 Binary strings (byte strings)

B<Binary strings>, or B<byte strings> are made of bytes. Here, you don't have
characters, just bytes. All communication with the outside world (anything
outside of your current Perl process) is done in binary.

On a binary string, you would do things like:

my (@length_content) = unpack "(V/a)*", $binary;
    $binary =~ s/\x00\x0F/\xFF\xF0/;  # for the brave :)
    print {$fh} $binary;
    my $byte_count = length $binary;

=head3 Encoding

B<Encoding> (as a verb) is the conversion from I<text> to I<binary>. To encode,
you have to supply the target encoding, for example C<iso-8859-1> or C<UTF-8>.
Some encodings, like the C<iso-8859> ("latin") range, do not support the full
Unicode standard; characters that can't be represented are lost in the
conversion.

=head3 Decoding

B<Decoding> is the conversion from I<binary> to I<text>. To decode, you have to
know what encoding was used during the encoding phase. And most of all, it must
be something decodable. It doesn't make much sense to decode a PNG image into a
text string.

=head3 Internal format

Perl has an B<internal format>, an encoding that it uses to encode text strings
so it can store them in memory. All text strings are in this internal format.
In fact, text strings are never in any other format!

You shouldn't worry about what this format is, because conversion is
automatically done when you decode or encode.

=head2 Your new toolkit

Add to your standard heading the following line:

use Encode qw(encode decode);

Or, if you're lazy, just:

use Encode;

=head2 I/O flow (the actual 5 minute tutorial)

The typical input/output flow of a program is:

1. Receive and decode
    2. Process
    3. Encode and output

If your input is binary, and is supposed to remain binary, you shouldn't decode
it to a text string, of course. But in all other cases, you should decode it.

Decoding can't happen reliably if you don't know how the data was encoded. If
you get to choose, it's a good idea to standardize on UTF-8.

my $foo   = decode('UTF-8', get 'http://example.com/');
    my $bar   = decode('ISO-8859-1', readline STDIN);
    my $xyzzy = decode('Windows-1251', $cgi->param('foo'));

Processing happens as you knew before. The only difference is that you're now
using characters instead of bytes. That's very useful if you use things like
C<substr>, or C<length>.

It's important to realize that there are no bytes in a text string. Of course,
Perl has its internal encoding to store the string in memory, but ignore that.
If you have to do anything with the number of bytes, it's probably best to move
that part to step 3, just after you've encoded the string. Then you know
exactly how many bytes it will be in the destination string.

The syntax for encoding text strings to binary strings is as simple as decoding:

$body = encode('UTF-8', $body);

If you needed to know the length of the string in bytes, now's the perfect time
for that. Because C<$body> is now a byte string, C<length> will report the
number of bytes, instead of the number of characters. The number of
characters is no longer known, because characters only exist in text strings.

my $byte_count = length $body;

And if the protocol you're using supports a way of letting the recipient know
which character encoding you used, please help the receiving end by using that
feature! For example, E-mail and HTTP support MIME headers, so you can use the
C<Content-Type> header. They can also have C<Content-Length> to indicate the
number of I<bytes>, which is always a good idea to supply if the number is
known.

"Content-Type: text/plain; charset=UTF-8",
    "Content-Length: $byte_count"

=head1 SUMMARY

Decode everything you receive, encode everything you send out. (If it's text
data.)

=head1 Q and A (or FAQ)

After reading this document, you ought to read L<perlunifaq> too.

=head1 ACKNOWLEDGEMENTS

Thanks to Johan Vromans from Squirrel Consultancy. His UTF-8 rants during the
Amsterdam Perl Mongers meetings got me interested and determined to find out
how to use character encodings in Perl in ways that don't break easily.

Thanks to Gerard Goossen from TTY. His presentation "UTF-8 in the wild" (Dutch
Perl Workshop 2006) inspired me to publish my thoughts and write this tutorial.

Thanks to the people who asked about this kind of stuff in several Perl IRC
channels, and have constantly reminded me that a simpler explanation was
needed.

Thanks to the people who reviewed this document for me, before it went public.
They are: Benjamin Smith, Jan-Pieter Cornet, Johan Vromans, Lukas Mai, Nathan
Gray.

=head1 AUTHOR

Juerd Waalboer <#####@juerd.nl>

=head1 SEE ALSO

L<perlunifaq>, L<perlunicode>, L<perluniintro>, L<Encode>

cmd: