Outline

• More On Yioop
• In-Class Exercise
• PHP

Introduction

• On Monday, we looked at the Yioop search engine which is coded in PHP.
• We will finish up our first quick overview of Yioop in a moment.
• This semester we will make use of the Yioop search engine as an information retrieval library from time to time in this class.
• In order to understand how to do this, we also need to know how to write a little PHP.
• So today, we are also going to give a brief tutorial on how to do this.

Configuring Your Crawl

• Yioop supports numerous options for controlling what sites you crawl and how you crawl them.
• The relevant activities for controlling these can be found under the Crawls activities group:
  
  • Manage Crawl is used for running crawls, managing existing crawls, and specifying what sites crawls are allowed to go to.
  • Manage Classifiers is used to create web page classifiers using existing crawls, these classifiers can then be used to label web pages during new crawls
  • Page Options is used to control what file types can be indexed and how data is extracted from a crawled page.
  • Results Editor is used to control what sites are allowed to appear in search results from an existing crawl, for example, preventing a site from appearing after a take down notice.
  • Search Sources is used to specify urls which should be periodically updated as news pages as well as podcasts that should be periodically downloaded. Periodic updating of pages only occurs if the MediaUpdater process is turned on under Manage Machines.
  • Web Scrapers allow one to specify a signature to detect a kind of web page as well as preprocessing instructions for how to handle that kind of web page in terms of how to locate content and meta words to inject. For example, one can make a Web Scraper specifically designed to get the important content out of a Wordpress site or to add a media:video meta word to appropriate Youtube pages, etc.

Crawl Options

• By default, Yioop crawls a collection of web sites I thought were popular.
• If one goes to Manage Crawls and clicks the Options link next to the Start New Crawl button one should see:
  
• The tabs at the top determine if Yioop will do a web crawl (default) or a local crawl of some text repositories such as a Wikipedia Dump.
• Here are some of the important options on the Web Crawl tab:
  • Crawl Order specifies what kind of queue is used to decide what to crawl next. The two choices are Page Importance, or Breadth-first.
  • The Get Crawl Options From drop down lets you load option from previous crawls you have done.
  • Checking the Restrict Sites By Url checkbox will display another textarea in which you can enter the only allowed urls for the next crawl to download.
  • Disallowed Sites/Sites with Quotas disallowed sites can be used to specify pages that Yioop is not allowed to download or to rate limit some sites.
  • Seed Sites is the list of urls the crawl will start from.

Page Options

• The Page Options activity has three tabs:
  
• The Crawl Time tab is used to control the processing of documents at crawl time. For example: What file extensions will be processed at all. How many bytes to use/download from a document. What summarizer should be run on a document and how many bytes output is it allowed to produce. What classifiers should be used. How long between crawls of a page. What classifiers should be used to label or rank documents. Which indexing plugins should be used. What operations in the Page Rules language should be applied.
• The Search Time tab is used to control at search time how search results are displayed.
• The Test Options tab is used how the options set above effect the processing of a given page. It has a text area that you can cut and paste the source code of a page into, and then run and get results of how Yioop would process the page.

Other Things Yioop Can Do

• Under the Social activities group:
  
  are four activities used in managing and view groups of users within Yioop.
• Admins and users can use Manage Groups to create and manage groups of other users at various access levels.
• Each group comes with its own discussion Feed and Wiki.
• On install, Yioop comes with three built in groups: Help, Public, and Search.
• These are configured by default to be only editable by the admin account.
• The Help group has the wiki pages for the built-in help system for Yioop.
• The Public wiki together with the Appearance activity in Yioop can be used to control the look of a Yioop web site. For example, the seekquarry.com website is a vanilla install of Yioop where this has been done to change its appearance.
• The Search Group is used to stored edited search results as well as wiki-callout search results. These can be configured under the Results Editor.
• The Configure activity is another useful activity. You can use it to turn on Query info and Error Info. These can be used to understand what iterators Yioop will use in processing a query.

Introduction to PHP

• PHP was developed by Rasmus Lerdorf around 1994-1995.
• PHP originally stood for Personal Home Page, later PHP became an acronym for PHP: Hypertext Processor.
• It is one of the most common server-side programming languages.
• The Wikipedia article on PHP says over 20 million domains are hosted on servers with PHP enabled.
• One oft-touted advantage of PHP is its shared nothing architecture: Each php request is distinct and does not directly share memory/data with other requests. Shared data is stored in filesystem/memory cache/database/etc.
• This can make PHP easier to debug and easier to scale.

Getting and Running PHP

• The web site for PHP is http://www.php.net/
• PHP can either be run from the command-line, run through a web server module, or fast-cgi.
• When run from a web server, the server determines that a page is PHP usually by looking for one of the extensions .php, .php3, .phtml
• When executing a PHP page, the PHP processor takes a PHP document and starts processing in one of two modes.
• By default, it simply copies it to the output. (Copy mode).
• When it sees a start PHP indicator (using <?php or <?=) it executes code and writes the result to the output until it sees a stop PHP indicator (?>)(Interpretive mode).
• A stop PHP indicator switches processing back into copy mode.
• PHP is typically interpreted but it also can be precompiled.
• The syntax of PHP is closely related to Perl, Javascript, and Java.

Example PHP file

Consider the file test.php:

hello world
<?php
echo "hello world";
?>
yeah!!
<?php
echo "okay";

If we ran this from the command line using:

php test.php

This would output:

hello world
hello worldyeah!!
okay

More On PHP

• PHP proceeds top-down when going through a file. Single statements in global scope are just executed.
• In global scope, we can also define new functions and classes.
• These can then be used in later global scope statements.
• For example,

  <?php
  function foo()
  {
     echo "hi";
  }
  foo(); // prints hi
  

• PHP can use either Perl, C++, or C comments: #, //, /* ... */

In-Class Exercise

• Install PHP.
• Make a file test.php which output "Hello," using copy mode, then calls a function yourName(); (after defining it).
• Implement the function yourName() so that it echo's your actual name.
• Please post your solutions to the Feb 10 In-Class Exercise Thread.

Configuring PHP

• You can fine-tune how PHP runs by editing the file php.ini
• A ';' is used to indicate a comment in this file.
• One common thing to change is the amount of error reporting PHP gives.
• By default, it is assumed we are in a production environment so none is given.
• For today, we want error-reporting. So find the line:

  ;error_reporting = E_ALL & ~E_NOTICE
  

  and uncomment it, restart Apache

PHP configuration

You can see what PHP's configuration settings are in use by the following small program

<?php
phpinfo();

Primitives and Variables

• PHP has four scalar types: boolean, integer, double, and string.
• There are also two compound types: array and object and two special types: resource and null.
• PHP is dynamically typed: you don't need to specify the types of variables before using them.
• Variable names are alphanumeric+underscore strings beginning with a dollar sign: $my_var, $fooFOO1999, etc. The first character after the $ is not allowed to be numeric.
• Yioop coding guidelines are to use variables with meaningful names where words are separated by underscores.
• PHP does variable assignments like most languages:

  $a = 53;
  $b = $a + 1;
  

• An unassigned variable is sometimes called an unbound variable and has value null. This can be coerced to one of the other types depending on the context.
• To check if a variable is currently bound you can use the isset($variable_name) function. This returns true or false (one of the two Boolean values, case-insensitive).
• empty($variable_name) can be used for isset($variable_name) && $variable_name != null.
• You can also call error_reporting(-1), to set PHP's error reporting level to report all errors, warnings, notices, etc. so that it prints unbound variables.
• PHP also has useful functions for printing out the values of variables: var_dump, print_r.