Outline

• PHP
• Quiz

Introduction

• On Wednesday, we looked at the Yioop search engine which is coded in PHP.
• We will finish up our first quick overview of Yioop in a moment.
• This semester we will make use of the Yioop search engine as an information retrieval library from time to time in this class.
• In order to understand how to do this, we also need to know how to write a little PHP.
• So today, we are going to give a brief tutorial on how to do this.

Configuring Your Crawl

• Yioop supports numerous options for controlling what sites you crawl and how you crawl them.
• The relevant activities for controlling these can be found under the Crawls activities group:
  
  • Manage Crawl is used for running crawls, managing existing crawls, and specifying what sites crawls are allowed to go to.
  • Manage Classifiers is used to create web page classifiers using existing crawls, these classifiers can then be used to label web pages during new crawls
  • Page Options is used to control what file types can be indexed and how data is extracted from a crawled page.
  • Results Editor is used to control what sites are allowed to appear in search results from an existing crawl, for example, preventing a site from appearing after a take down notice.
  • Search Sources is used to specify urls which should be periodically updated as news pages as well as podcasts that should be periodically downloaded. Periodic updating of pages only occurs if the MediaUpdater process is turned on under Manage Machines.
  • Web Scrapers allow one to specify a signature to detect a kind of web page as well as preprocessing instructions for how to handle that kind of web page in terms of how to locate content and meta words to inject. For example, one can make a Web Scraper specifically designed to get the important content out of a Wordpress site or to add a media:video meta word to appropriate Youtube pages, etc.

Crawl Options

• By default, Yioop crawls a collection of web sites I thought were popular.
• If one goes to Manage Crawls and clicks the Options link next to the Start New Crawl button one should see:
  
• The tabs at the top determine if Yioop will do a web crawl (default) or a local crawl of some text repositories such as a Wikipedia Dump.
• Crawl Order specifies what kind of queue is used to decide what to crawl next. The two choices are Page Importance, or Breadth-first.
• The Get Crawl Options From drop down lets you load option from previous crawls you have done.
• Checking the Restrict Sites By Url checkbox will display another textarea in which you can enter the only allowed urls for the next crawl to download.
• Disallowed Sites/Sites with Quotas disallowed sites can be used to specify pages that Yioop is not allowed to download or to rate limit some sites.
• Seed Sites is the list of urls the crawl will start from.

Page Options

• The Page Options activity has three tabs:
  
• The Crawl Time tab is used to control the processing of documents at crawl time. For example: What file extensions will be processed at all. How many bytes to use/download from a document. What summarizer should be run on a document and how many bytes output is it allowed to produce. What classifiers should be used. How long between crawls of a page. What classifiers should be used to label or rank documents. Which indexing plugins should be used. What operations in the Page Rules language should be applied.
• The Search Time tab is used to control at search time how search results are displayed.
• The Test Options tab is used how the options set above effect the processing of a given page. It has a text area that you can cut and paste the source code of a page into, and then run and get results of how Yioop would process the page.

Other Things Yioop Can Do

• Under the Social activities group:
  
  are three activities used in managing and view groups of users within Yioop.
• Admins and users can use Manage Groups to create and manage groups of other users at various access levels.
• Each group comes with its own discussion Feed and Wiki.
• On install, Yioop comes with two built in groups: Public and Help.
• These are configured by default to be only editable by the admin account.
• The Public wiki together with the Appearance activity in Yioop can be used to control the look of a Yioop web site. For example, the seekquarry.com website is a vanilla install of Yioop where this has been done to change its appearance.
• The Configure activity is another useful activity. You can use it to turn on Query info and Error Info. These can be used to understand what iterators Yioop will use in processing a query.

Introduction to PHP

• PHP was developed by Rasmus Lerdorf around 1994-1995.
• PHP originally stood for Personal Home Page, later PHP became an acronym for PHP: Hypertext Processor.
• It is one of the most common server-side programming languages.
• The Wikipedia article on PHP says over 20 million domains are hosted on servers with PHP enabled.
• One oft-touted advantage of PHP is its shared nothing architecture: Each php request is distinct and does not directly share memory/data with other requests. Shared data is stored in filesystem/memory cache/database/etc.
• This can make PHP easier to debug and easier to scale.

Getting and Running PHP

• The web site for PHP is http://www.php.net/
• PHP can either be run from the command-line or run through a web server module.
• When run from a web server, the server determines that a page is PHP usually by looking for one of the extensions .php, .php3, .phtml
• When executing a PHP page, the PHP processor takes a PHP document and starts processing in one of two modes.
• By default, it simply copies it to the output. (Copy mode).
• When it sees a start PHP indicator (using <?php or <?=) it executes code and writes the result to the output until it sees a stop PHP indicator (?>)(Interpretive mode).
• A stop PHP indicator switches processing back into copy mode.
• PHP is typically interpreted but it also can be precompiled.
• The syntax of PHP is closely related to Perl, Javascript, and Java.

Example PHP file

Consider the file test.php:

hello world
<?php
echo "hello world";
?>
yeah!!
<?php
echo "okay";

If we ran this from the command line using:

php test.php

This would output:

hello world
hello worldyeah!!
okay

More On PHP

• PHP proceeds top-down when going through a file. Single statements in global scope are just executed.
• In global scope, we can also define new functions and classes.
• These can then be used in later global scope statements.
• For example,

  <?php
  function foo()
  {
     echo "hi";
  }
  foo(); // prints hi
  

• PHP can use either Perl, C++, or C comments: #, //, /* ... */

Configuring PHP

• You can fine-tune how PHP runs by editing the file php.ini
• A ';' is used to indicate a comment in this file.
• The most common thing for people to change is to have a line:

  short_open_tag = On
  

  so that you can escape to a PHP script using <? rather than <?php
• Unfortunately, using the short open tag can mean your script is not as portable so for this class we will leave this off.
• Another common thing to change is the amount of error reporting PHP gives.
• By default, it is assumed we are in a production environment so none is given.
• For today, we want error-reporting. So find the line:

  ;error_reporting = E_ALL & ~E_NOTICE
  

  and uncomment it, restart Apache

PHP configuration

You can see what PHP's configuration settings are in use by the following small program

<?php
phpinfo();

Primitives and Variables

• PHP has four scalar types: boolean, integer, double, and string.
• There are also two compound types: array and object and two special types: resource and null.
• PHP is dynamically typed: you don't need to specify the types of variables before using them.
• Variable names are alphanumeric+underscore strings beginning with a dollar sign: $my_var, $fooFOO1999, etc. The first character after the $ is not allowed to be numeric.
• Yioop coding guidelines are to use variables with meaningful names where words are separated by underscores.
• PHP does variable assignments like most languages:

  $a = 53;
  $b = $a + 1;
  

• An unassigned variable is sometimes called an unbound variable and has value null. This can be coerced to one of the other types depending on the context.
• To check if a variable is currently bound you can use the isset($variable_name) function. This returns true or false (one of the two Boolean values, case-insensitive).
• empty($variable_name) can be used for isset($variable_name) && $variable_name != null.
• You can also call error_reporting(-1), to set PHP's error reporting level to report all errors, warnings, notices, etc. so that it prints unbound variables.
• PHP also has useful functions for printing out the values of variables: var_dump, print_r.

Integer, String, and Boolean Type

• Integers in PHP correspond to C long's (signed), so their size depends on the size of a long on a given machine.
• This is usually 32 bits.
• PHP's double type corresponds to C's double and literals follow the format for C floating point literals: I.e., .345, 3E-7, etc.
• Characters in PHP are single bytes (No Unicode! -- we'll talk about how to fake later).
• String literals are built up out of these characters.
• PHP distinguishes between single quote and double quote literals in the same way that Perl does. So the variable in "Bob=$bob" is interpolated but in 'Bob=$bob' is not.
• There are only two values for Boolean's: true or false. As for coercion, the empty string or a "0" or a "0.0" is interpreted as FALSE. Otherwise, strings evaluate to TRUE. Integers and double evaluate to TRUE as long as they aren't zero.

Arithmetic Operators,etc

• PHP supports the usual arithmetic operators: +, -, *, /, %, ++, --.
• PHP also does type coercion.
• Division should be assumed to output a double if any fractional value exists.
• Some useful predefined functions are: floor, ceil, round, srand, rand, abs, min, and max.
• String concatenation is done with a period: "string1". "string2" is "string1string2".
• Some useful string operations are strlen, strcmp, strpos, substr, chop, trim, ltrim, strtolower, strtoupper.
• If you are doing either string or array manipulation, you should check the PHP manual's String Functions and Array Functions to see if there is a function that already does what you want. Using a built in function will be much faster.
• Type conversion can be done using expressions like (int)$sum or intval($sum) or settype($sum, integer).
• You can also check the type with is_int, is_integer, is_long, etc.
• Assignment operators are like in C, Java, etc.

Control Statements

• PHP supports: if, elseif, else if, else, switch/case: For example,

  //if
  if($a) print "hello";
  //switch case:
  switch($a)
  {
     case 5:
        echo "hello";
        break;
  }
  

• for:

  for($a = 0; $a<10; $a++){echo "hello $a";}
  

• while:

  while(!$var) { /*do something*/}
  

• do while:

  do {/*do something */} while (!$var);
  

Arrays

• Arrays can be declared with the syntax:

  $a = array("hi", 1, 2);
  //for PHP >= 5.4 you can also use the syntax:
  $a = ["hi", 1, 2]; //what to use in all modern code
  

• You can nest arrays:

  $b = ["hi", [1,2], 2];
  $c = ["a" => "c"]; // key => value associative array
  

• You can dereference arrays as in most languages:

  echo $a[0]." ".$c['a']; //$c['a'] returns "c"
  

• You can cycle over elements of an array using foreach:

  foreach($arr as $var) {echo $var;}
  
  foreach($assoc_arr as $key => $value) {echo "Key $key Value $value";}
  
  

More on PHP Arrays

• All arrays in PHP can be viewed as associative arrays...
• A complicated way to write out the array [1, 2, 3] is:

  [0=>1, 1=>2, 2=>3]
  

• To get an array of keys or an array of values we can use the functions:

  $keys = array_keys($arr) and $values = array_values($arr);
  

• Arrays can also be created by an assignment:

  $barr[1] = 5; // creates array $barr if doesn't exist
  

• If we did the assignment $barr[] = 6; then since the argument to [] wasn't specified PHP will assign $barr[2] = 6;

Yet More on PHP Arrays

• You can call the unset function on the element in an array: $list= [2,4,6,8]; unset($list[2]);
• Some useful array functions: count -- returns the number of elements in an array, is_array, in_array, implode, explode, sort.
• To see how implode/explode work consider:

  $str="this is a string";
  $words = explode(" ", $str); /*acts like split except here the first
   argument is a string rather than a regular expression. 
   So words is an array("this", "is", "a", "string"). PHP has a split function 
   but not as fast, since arg might be a regular expression. */
  $str2 = implode(" ", $words); //undoes the explode. 
  

Functions

• The general format of a PHP function is:

  function name([parameter list]){...}
  

• For example,

  function inc($i){return ++$i;} 
  

• A return value can be sent back using a return call as in many programming languages.
• You can modularize your code by putting several function definitions into a file and then use the include function to include them into any document that needs those functions.
• Parameters are passed by value. So the function call:

  $b = inc($a); // leaves the value of $a unchanged
  

• You can also create functions with pass by reference parameters:

  function inc(&$i){...}
  

Quiz

Which of the following is true?

1. Zipf's law for terms in a corpus says that the frequency of the `i`th most common term, `F_i`, will be proportional to `1/(i^\alpha)` for some constant `\alpha`.
2. Every Markov Model is equivalent to a first-order language model.
3. Yioop uses the same process to schedule what to crawl next as it does to download web pages.

Variable Scope

• The default scope of a variable in PHP is only within the function that it is used. That is local scope:

  $bob = 5;
  function test()
  { 
     $bob = 6; echo $bob; //echo's 6
  }
  test();
  echo $bob; //echo's 5
  

• In order to access global variables within a local function one would need to use the global declaration:

  $bob = 5;
  function test()
  {
     global $bob; # if did not do bob would be NULL 
  
     echo $bob; 
  }
  test();
  

  PHP also supports static local variables. These preserve states between function calls:

  function addone () {static $count =0; echo $count++;}
  

Built-in Globals

• PHP makes available several important global variables which are useful for server side scripts.
• The phpinfo() function can be used to find out all of these globals.
• Except for $argv which has just global scope, the rest of super-globals and are available in any scope.
• Here are some of the main ones:

  $argv

  An array of the command-line arguments when a script is run from the command line.

  $_SERVER

  an array of information about the server like $_SERVER["SERVER_NAME"], $_SERVER ["DOCUMENT_ROOT"], $_SERVER ["QUERY_STRING"], etc

  $_ENV

  an array of info about the runtime environment: $_ENV["PATH"], $_ENV["PWD"], etc.

  $_REQUEST

  an array of the variables that have been get'd or post's from forms. So if $_SERVER["QUERY_STRING"] was hi=there&hi2=there2 would have $_REQUEST["hi"] == "there" and $_REQUEST["hi2"]="there2";

  $_GET

  like $_REQUEST but only for get'd variables

  $_POST

  like $_REQUEST but for post'd variables

• PHP can be configured with register_globals = On, in which case the variable $hi would be a global with value "there". This is a bit risky security-wise.

File Reading

• To get an array of file which meet some pattern we can use the glob function:

  $foo_files = glob("./foo*"); /* return an array of file names in the current directory
                                  that begin with foo */
  

• To open a file one can do:

  $fileHandle = fopen("my.dat", "r"); // r is for reading, w is for writing, etc 
  $file_string = fread($fileHandle, filesize("my.dat")); //there is an fwrite function as well
  fclose($fileHandle);
  

  Here fread reads in its second parameter many bytes.
• To read in a single line from a file one can use:

  $line = fgets($fileHandle, $max_num_bytes_line); //this is an fputs function as well
  

• Alternatively, one can read and write the whole file as a string using a single commands like:

  $string = file_get_contents("my.dat");
  $string .= "foo";
  file_put_contents("my.dat", $string);
  

• Sometimes it is useful to read in the whole file as an array of lines. This can be done with:

  $lines = file("my.dat");