gettext

We start today by finishing up our discussion of localization and internationalization begun last week.

We next look at how we could set up the development environment of our web project to allow for text from more than one locale.
We will first look at how to manage static text.
One common tool to use to do this is gettext.
There are also some open source tools such as Pootle with web interfaces for managing the kind of files used with gettext.
The basic idea of gettext is that the developer, whenever they need to output a string that might need to be localized, calls a function gettext() with a string which says what needs to be translated. For example,
```
gettext("translate_me"); // you could also use _("translate_me");
```
The function gettext uses the current locale, textdomain, bindtextdomain to locate a .mo binary file which contains a string to echo for "translate_me" for that locale.

So you might have in your html file commands like:

putenv('LC_ALL=en_US');
setlocale(LC_ALL, 'en_US'); // say locale
bindtextdomain("myMoFile", "./locale"); // say locale dir
textdomain("myMoFile"); // say .mo file
_("translate_me"); // looks for ./locale/en_US/LC_MESSAGES/myMoFile.mo
                   // to find the translation.

How do we create .mo files? .po files

There are a few steps to create a .mo file for a given locale.
First, we have to extract all the things we need to translate from a given file.
To do this we use the command xgettext:
```
xgettext -L PHP filename.php
```
This will generate a messages.po file. If we want, myMoFile.po we could have done: -d myMoFile
At the top of this file there is a place to set the text encoding. Make sure to set this to UTF-8!

Inside this file we have lines like:

#: filename.php:6
msgid "translate_me"
msgstr ""

You would give this file to your localizers to fill in a msgstr to translate each msgid.

From .po files to .mo files, more on msgid's.

Once we have a fully translated .po file we are ready to generate a .mo file from it.
To do this we use a command like:
```
msgfmt -o myMoFile.mo myMoFile.po
```
We then place the .mo file in the correct directory and we have one file translated to one locale.
It can become hard to maintain if you use msgid's which are just English strings.
It would be bad if several files in your project had the same msgid "Next"; and where next might be translated in different ways to a locale depending on context.
Instead, you should a naming convention for id's which specifies things like: namespace_pagename_name_additional .
gettext has support for things like pluralizations, and messages that involve formatted text that I will not go into.

Dynamic String Translation

gettext has the advantage that it is fast.
Unfortunately, it doesn't work well in situations where you are dynamically adding new things to your application which need to be translate.
For instance, you might have a web form through which employees add new products and descriptions.
These descriptions still need to be translated.
To handle this, you probably would need to go to a slower approach involving using an SQL database.
You might have a table of Product with a column ID as a key for what is to be translated and a nother table Locale which have a LOCALE_ID and LOCALE_STRING (ex: en_US).
Finally, you could use a table like with columns like: ID, LOCALE_ID, TRANSLATION to store the given locale's translation of product information.

Quiz

Which of the following statements is true?

You can use a X-Frame-Options response header to prevent XSS web-site attacks.
All HTTP/1.1 protocol communication is encrypted using SSL.
The writing-mode css property can be used to display vertically aligned text.

General Coding Comments

In most settings, you will need to deploy your app at least twice -- once on your own machine, once on your deployment machine. You want your code to be as portable as possible.
You want to catch errors which can trip up deployment on various platforms and minimize time to understand and integrate new code if multiple people are working on your project.
So follow a set of coding guidelines for each language used in your project. For example, you can use the coding guidelines I developed for my own projects. For PHP, this boils down to: PSR-2.
Make sure every method you write is commented. Use PHPDocs for PHP.
Do not be schizophrenic in how you name variables/functions/methods/classes. Follow the convention of your coding guidelines.
When coding (not deploying to production) in PHP make sure you have error reporting set to the highest level (-1). You can check this by making a small file like:
```
<?php $blah++; ?>
```
You should see a notice saying blah is undefined.
Test on at least two platforms. For example, Windows and OSX or Windows and Linux.
Make sure your code does not generate any notices, warnings, or errors.
Do not be sloppy initially and hope that you will go back later to fix issues -- you almost never have the time/energy to go back later. If you are aware you are doing something sub-optimally, re-code it then and there.

PHP Guidelines

Avoid require or include commands within methods or randomly in your PHP documents. Put them at the top of the given file, try to use require_once, and try to avoid using variables to say what to require.
Do not write much code out of a function or method. If you are using the MVC approach we've been discussing for the HW's then only index.php and config.php should have code that is not inside a function/method/class.
In particular, use PHP's copy mode only within a function, and don't have a ?> at the end of a file.

As an example, consider the following code for a Layout class suitable for rendering the boiler plate start and end html tags:

<php
class Layout
{
    public $view;

    function __contruct($view)
    {
      $this->view = $view;
      //contructor code
    }
    function render($data)
    {
      ?>
      <html>
      <head><title><?=$data['title']; ?></title></head>
      <body>
      <?php $this->view->render($data); ?>
      </body>
      </html>     
      <?php
    }
}

This avoids headaches such as sending HTTP headers from a sessions after some HTML has been output.

Make sure to de-reference associative arrays correctly:

echo $my_assoc['my_field'];
//not
echo $my_assoc[my_field];

Always use Unix paths not Windows paths -- on a Windows machine PHP auto-converts from the former to the latter. On a *nix machine though, PHP won't convert Windows paths to *nix.

Testing

Write unit tests for your code, especially of any data structure classes you have. Use PHPT, SimpleTest, or PHPUnit.
To automate testing of UI use: Selenium or PhantomJS.

Scaling a Web-site

Today we will look at what's needed to move your site from handling a few thousand hits a day to thousands of hits per second.
Rough traffic (2008) - addons.mozilla.com (3000-6000 hits/second); wikipedia.org, a top ten site (30,000 -40,000 hits/second). (I know this is old, but I am reasonably sure of the numbers, and would be roughly ballpark for a large website.)
Scaling is a term that is often used in a vague sense.
To make this term precise, we will say a web-site is scalable if:
- It can accommodate increased usage
- It can accommodate an increased dataset
- It is maintainable.
Throwing more hardware at a problem might help solve the first two points, but if the additional hardware makes the system to hard to manage three will fail.
Let's look at an example web-site architecture in terms of scalability.

Example Web-site Architecture

Organization of a website show squids, http servers, memcached server, databases.

Remarks on Architecture

The diagram on the last slide only shows one master DB and many slaves.
The slaves would be read only and the master usually supports reads and writes. The slaves could be kept in sync with the master using rsync.
To handle more data might have several master DBs and horizontally partition the data between them.
We will mainly look-at how to handle increase usage.
The reverse proxy in front of everything is typically implemented using Varnish, Squid, or a hardware specific system.
An advantage of a layer seven network appliance like a Netscaler, etc hardware over a Squid is that it can do SSL in hardware. On the other hand, hardware is harder to change. Many products nowadays run virtualized and cloudified as well.
We will focus today on how to handle increased database traffic where most of the traffic comes in the forms of reads.

Content Delivery Networks

One problem with even an architecture like that provided a couple slides back, is that traffic all still has to come to your subnetwork of the internet and this could lead to a bottleneck.
To scale so that geographic effects don't mess with your ability to provide content, you often would also use a content delivery network.
Akamai is one early example of such a network. There is also Cloudflare. Amazon has Cloudfront and Microsoft has Azure.
One formerly popular, example of a free CDN was the Coral CDN. Unfortunately, it now seems to be defunct.
To cache a web page,
```
http://www.cs.sjsu.edu/faculty/pollett/
```
in this network, you used to be able to make a request to:
```
http://www.cs.sjsu.edu.nyud.net/faculty/pollett/
```
i.e., I tack on .nyud.net to the domain name.
CDN's are often implemented using a distributed hash table.
Like a usual hash table, such a table stores key value pairs.
Unlike a hash table, there is also a distance function d(k1, k2) which can be used to compare the distance between two keys.
A key might contain information about a url, an ip address (geographic info), and a replication number; a value might be a web page.
Each node in the network also has a key associated with itself.
Nodes in such a DHT network usually are arranged in some kind of topology like a ring with additional edges to keep the number of hops between any two nodes logarithmic.
When a request comes in to one node in this network, it checks if any neighbor has distance closer to the key, if so it let's the closest neighbor try to handle the request; otherwise, it handles it.

Object Relational Mappings

If our web-site is built using the MVC pattern that we have been using on the homeworks, then one of the common sources of database queries will be marshaling data for our models.
Let's briefly consider how this is done.
Typically, models are used to get objects into and out of a database.
As an example, a user might make many posts to a discussion board.
This is an example of a one-to-many relationship.
We might have two database tables user and post. To represent this relationship the post table might have a column uid as a foreign key reference into the user table.
Now a User object might be modeled as PHP associative array consisting of the name, and other fields of the user, one of these fields being an array of posts objects:
```
Array( "name" => "Bob", ..., "posts" => Array([0] => Array("title" => "Dark and Stormy Night", ...), [1]=>...))
```
An object relational mapping is a mapping between these kind of objects and database tables.
Besides one-to-many relationship, one might need to handle many-to-many, belongs-To, and one-to-one relationships.
In pretty much all cases, marshaling an object requires several database queries, which can be slow, so may be sped up by caching.

Memcached

Memcached (http://memcached.org/) allows you to create an in memory cache of data.
A memcache can be on a separate server from either your database or your webserver.
Your web app connects to the memcache daemon and checks if an item such as an object that might have been obtained from a database using an ORM mapping, is in the memcache. If it is you can get them quickly, if it isn't then you go to the database.
In PHP you can access a memcache daemon using the PECL extension:
http://pecl.php.net/package/memcache.
To install this you type
```
pecl install Memcache
```
and then add the line
```
extension=memcache.so
```
(or .dll on Windows) to your php.ini file and restart everything.

Running memcached

To run the memcached daemon one can type a line like:
```
memcached -d -m 50 -l 127.0.0.1 -p 11211
```
Here:
- -d -- as daemon
- -m -- number of megabytes
- -l -- location
- -p -- port
The original versions of memcached didn't have any security protocol, so the -l option is important -- it is allowing only traffic from the loopback network to connect to memcached.

Demo of connecting to memcached as a human - set

Memcached has two protocols for interacting with it: a text-based and a binary format.
We will demo connecting to memcache using telnet and the text-based protocol.

To connect and do a set we could do:

telnet 127.0.0.1 11211
set foo 0 0 6
lalala
STORED

Here set is the command.
foo is the key which we can use to look up the value we stored later.
The first 0 is a 16 or 32 bit (1.2.1 or higher), decimal written flag which could be used to optionally store info.
The second zero indicates the expiry time in seconds -- 0 means never expire.
The 6 indicates the number of bytes we'll store not including the final carriage return linefeeds which terminate the data block.
After these value we have a \r\n and then we begin the data block which is terminated with a \r\n.
Finally, memcached responds with STORED.
Other storage commands include: add, replace, append, and prepend.
There are also commands: delete, to delete entries, incr, decr to increment decrement the value associated with a key, and stats to output various statistics.

Demo of connecting to memcached as a human - get

To retrieve stored values, you can use either get followed by one or whitespace separated keys, or gets followed by one or more whitespace separated keys.
The server will respond with:
```
VALUE key flags bytes\r\n
data block\r\n
```
one for each value returned, followed by an END\r\n after the last value.
Continuing the example from the last slide we can type and get the response:
```
get foo
VALUE foo 0 6
lalala
END
```

Some Remarks on Memcached

Using CURL it would be possible to write a pure PHP program that connects with a memcache to store/retrieve values.
Instead, people usually use one of two PHP extensions: Memcache (older) or Memcached written in C.
These extensions are used because they allow for persistent (between php requests) connections to a memcache.
Typically, memcache's are used as a distributed cache. If one has 50Mb free on each of ten machines, one adds the ten servers to the memcache objects in PHP and it manages balancing and storing the key-value amongst these machines. This effectively would give you 500MB of total memory storage.
As of 2008, Facebook was making use of 15TB of memcache storage. As of 2013/2014, Facebook was still using memcache, but I don't have any statistics on hom much.
Twitter uses a variant of Memcache called Fatcache.
The distributed nature of memcached and the fact that is language agnostic, distinguishes it from things like APC (the alternative PHP cache -- built-in to PHP) which provide both op-code caching and single machine object caching.
Memory in memcached is allocated using a slab system with slabs of various sizes. When a given pool of slabs fills an LRU algorithm, is used to determine who is bumped from memory.
It should be noted that there is a 1MB maximum on the size of an object you can store with a given key.
As one common use of memcached is to store objects from ORM, this limit usually does not cause a problem. The PHP extensions do the job of serializing/unserializing these objects before putting them in/retrieving them from a memcache.

Using memcached in PHP

To use memcached within PHP one needs to create a memcache object:
```
$memcache = new Memcache();
```

Next one needs to set-up the memcache server:

$memcache->addServer($host, $port, $persistent, $weight, $timeout, $retry_interval);

To add something to the cache one could use a command like:

$memcache->set($key, $val, $flag, $expire); 
    // $flag is whether to compress the item in the store
    //$expire - Unix timestamp, or seconds from present. 0 -don't expire

To read from the cache one can use a line like:
```
       
$memcache->get($key);
```

Example Memcache Test

<html>
<head><title>Memcache Test</title></head>
<body>
<h1>Test</h1>
<?php
  $memcache = new Memcache();
  $memcache->addServer("localhost", 11211, true, 1, 1, 5);
  $memcache->set("bob", "value", null, 60);
  echo "<p>".$memcache->get("bob")."</p>";
?>
</body>
</html>

Finish L10n, Scalability

Outline