Outline

• Getting Started With Yioop
• Yioop Prerequisites and Installation
• Performing a Web Crawl
• Quiz

Yioop

• Yioop is an open-source PHP search engine/indexer/crawler/IR library/wiki discussion board/ad platform created by me and some of master's students beginning in the Fall of 2009.
• Details of when different features were added can be found in the Changelog.
• Yioop can be downloaded from: https://www.seekquarry.com/p/Downloads
• Alternatively, you can get various versions from the ViewGit Web interface to the Yioop's git repository or by cloning the most recent version using:

  git clone https://seekquarry.com/git/yioop.git
  

• You probably want to do the latter for this class.
• It is highly recommended to briefly read the online documentation for Yioop.
• To run Yioop you need: PHP 7.4 or greater with at least PDO sqlite support and compiled with the cURL, multi-byte string, libxml, bcmath, and GD libraries.
• Yioop comes with/makes use of a single file Node/React-esque PHP WebServer/Router which may be independently interesting or useful for other projects.

Yioop Prerequisites if Running Under an Existing Web Server

• Yioop can also be run under an existing web server, in which case, that web server should enable URL rewriting.
• The version of Apache and PHP that come built-in to a Mac already support this.
• In Ubuntu Linux (tested 20.04LTS), the following sequence of commands will get you what you need:

   sudo apt install curl
   sudo apt install php-cli
   sudo apt install php-mbstring
   sudo apt install php-sqlite
   sudo apt install php-curl
   sudo apt install php-gd
   sudo apt install php-xml
   sudo apt install php-bcmath
  

• Alternatively, on Windows/Mac/Linux you could install Xampp. Just make sure to enable the php_curl.dll or php_curl.so extension in the php.ini file.

Configuration

• Move the downloaded version of Yioop under your web server's DocumentRoot.
• Yioop has a subfolder called work_directory. Make sure Yioop has read/write access to this directory.
• Now point your browser to the location on localhost where you put Yioop. With any luck you should see a screen like:
  
• Congratulations, you have successfully installed Yioop!

Logging in

• To make a web crawl you need to login at this point. Click on the hamburger menu and choose Signin.
• The default username is root and the password field should be left blank.
• You should now see the default admin page:
  

Activities

• Things you can do are broken down into activities which are listed under the Hamburger menu:
  
• On login, you are on the Manage Account activity page. Here you can change your password if you like.
• On the Manage Account page there also icon based quick links to a variety of useful activities. For example, the buttons
  
  lead respectively to the Manage Crawls, Manage Machines, Crawl Mixes, and Query Statistics activities.
• Returning to the admin activities listed under the hamburger menu, the Configure activity allows you to turn on error reporting, query info, and unit test, so can be useful for debugging.
• Under Server Settings, you can control if accounts other than the root account can be created.
• Users can be managed under Manage Users.
• What activities a user has access to is controled by what roles they have (by default Admin or User). Roles consist of collection of allowed activities and can be managed under Manage Roles.
• You are now ready to crawl!

Scripts for Crawling

• Crawling involves two scripts, a QueueServer script and a Fetcher script, which are located in Yioop's src/executables directory.
• These scripts can be run either from within the web interface (requires that the web server has permission to schedule jobs) or from the command line.
• From the web interface to manually run these scripts go to Manage Machines, then click the "On" button for each of them.
  
• The speed of a crawl is determined by the number of web pages that can be simultaneously downloaded (adding more fetchers tends to increase this), how quickly the priority queue of what to crawl next can be updated, and how quickly the downloaded files can be indexed (the number of queue servers affects the latter two).
• A rule of thumb is that the queue server operates the most efficient if it can run on two CPU cores to itself and a fetcher operates most efficiently if it get a CPU core to itself.
• Under the Manage Machines activity you can modify the number of queue servers and fetcher one has for a crawl or you can leave them at the default settings by using the "+" button.
• If you want to run these scripts from the command line, open two command shells and in each go to Yioop's src/executables directory.
• In one type:

  php Fetcher.php terminal
  

  in the other type:

  php QueueServer.php terminal
  

  Be aware that for this to work the php command has to be in your path environment variable; otherwise, you need to give the full path to php.

Crawling Nice

• When a crawler or a web browser requests a web page it sends User-Agent info.
• This info can tell a site owner who to contact in case your crawler behaves badly.
• You can set this information for your crawler under the Configure activity:

Performing a Crawl

• Now go back to the admin panel of Yioop and click on Manage Crawls.
• If you type in a name of a crawl and click Start New Crawl, Yioop will start crawling with whatever the current crawl settings are.
• If no queue servers or fetchers are running, it will turn on one of each for you before it start.
• To see and modify the settings of what to crawl click the Options link next to start new crawl.
• The Options page largely controls what urls or documents will be crawled, to control how individual documents are processed click on the [Page Processing Options] link:
• The Test Options tab here let's you test how a single page will be crawled.
• Going back to crawling, once you have started a crawl, Yioop will keep crawling until it runs out of new URLs to crawl and it times out or until you click Stop Crawl.
• From the time you click Stop Crawl until your crawl is ready is not instantaneous so please be patient... Do not stop the queue servers or fetchers. When Yioop has properly closed the active crawl, it will appear in the Previous Crawls list on the manage crawls page. At this point it should be safe to stop the fetchers and queue servers, if you want.

Seeing Search Results

• Pick a crawl that is listed under Previous Crawls and click on its "Set as Index" link.
• Click on the Yioop logo.
• This should take you back to the Yioop search screen.
• Typing in a query on this screen should now yield results from the crawl you set as index.

Quiz

Which of the following is true?

1. Zipf's law says that the frequency of the ith most common term, `F_i`, will be proportional to `1/{i^\alpha}` for some constant `\alpha`.
2. The zero'th order language model considered in class, `M(sigma) = frac{mbox{frequency}(sigma)}{sum_{sigma' in V}mbox{frequency}(sigma')}` cannot be easily extended to allow for calculation the probability of phrases.
3. If a phrase like "fourth witch" had non-zero probability in a non-smoothed, first order language model, there is a setting of `\gamma` under which it could have zero probability in a smoothed model.