Outline

• Getting Started With Yioop
• Yioop Prerequisites and Installation
• Performing a Web Crawl
• Quiz
• Getting Help within Yioop
• Media Updater
• Feed Update Job

Yioop

• Yioop is an open-source PHP search engine/indexer/crawler/IR library/wiki discussion board/ad platform created by me beginning in the Fall of 2009.
• Details of when different features were added can be found in the Changelog.
• Yioop can be downloaded from: https://www.seekquarry.com/p/Downloads
• Alternatively, you can get various versions from the ViewGit Web interface to the Yioop's git repository or by cloning the most recent version using:

  git clone https://seekquarry.com/git/yioop.git
  

• It is highly recommended to briefly read the online documentation for Yioop.
• To run Yioop you need: PHP 7.2 or greater with at least PDO sqlite support and compiled with the cURL, multi-byte string, libxml, bcmath, and GD libraries.
• Yioop comes with/makes use of a single file Node/React-esque PHP WebServer/Router which may be independently interesting or useful for other projects.

Yioop Prerequisites if Running Under an Existing Web Server

• Yioop can also be run under an existing web server, in which case, that web server should enable URL rewriting.
• The version of Apache and PHP that come built-in to a Mac already support this.
• In Ubuntu Linux (tested 18.04LTS), the following sequence of commands will get you what you need:

   sudo apt install curl
   sudo apt install php-cli
   sudo apt install php-mbstring
   sudo apt install php-sqlite
   sudo apt install php-curl
   sudo apt install php-gd
   sudo apt install php-xml
   sudo apt install php-bcmath
  

• Alternatively, on Windows/Mac/Linux you could install Xampp. Just make sure to enable the php_curl.dll or php_curl.so extension in the php.ini file.

Configuration

• Move the downloaded version of Yioop under your web server's DocumentRoot.
• Yioop has a subfolder called work_directory. Make sure Yioop has read/write access to this directory.
• Now point your browser to the location on localhost where you put Yioop. With any luck you should see a screen like:
  
• Congratulations, you have successfully installed Yioop!

Logging in

• To make a web crawl you need to login at this point. Click on the hamburger menu and choose Signin.
• The default username is root and the password field should be left blank.
• You should now see the default admin page:
  

Activities

• Things you can do are broken down into activities which are listed under the Hamburger menu:
  
• On login, you are on the Manage Account activity page. Here you can change your password if you like.
• The Configure pages allows you to turn on error reporting, query info, and unit test, so can be useful for debugging.
• Under Server Settings, you can control if accounts other than the root account can be created.
• Users can be managed under Manage Users.
• What activities a user has access to is controled by what roles they have (by default Admin or User). Roles consist of collection of allowed activities and can be managed under Manage Roles.
• You are now ready to crawl!

Scripts for Crawling

• Crawling involves two scripts, a QueueServer script and a Fetcher script, which are located in Yioop's src/executables directory.
• These scripts can be run either from within the web interface (requires that the web server has permission to schedule jobs) or from the command line.
• From the web interface to manually run these scripts go to Manage Machines, then click the "On" button for each of them.
  
• The speed of a crawl is determined by the number of web pages that can be simultaneously downloaded (adding more fetchers tends to increase this), how quickly the priority queue of what to crawl next can be updated, and how quickly the downloaded files can be indexed (the number of queue servers affects the latter two).
• A rule of thumb is that the queue server operates the most efficient if it can run on two CPU cores to itself and a fetcher operates most efficiently if it get a CPU core to itself.
• Under the Manage Machines activity you can modify the number of queue servers and fetcher one has for a crawl or you can leave them at the default settings.
• If you want to run these scripts from the command line, open two command shells and in each go to Yioop's src/executables directory.
• In one type:

  php Fetcher.php terminal
  

  in the other type:

  php QueueServer.php terminal
  

  Be aware that for this to work the php command has to be in your path environment variable; otherwise, you need to give the full path to php.

Crawling Nice

• When a crawler or a web browser requests a web page it sends User-Agent info.
• This info can tell a site owner who to contact in case your crawler behaves badly.
• You can set this information for your crawler under the Configure activity:

Performing a Crawl

• Now go back to the admin panel of Yioop and click on Manage Crawls.
• If you type in a name of a crawl and click Start New Crawl, Yioop will start crawling with whatever the current crawl settings are.
• If no queue servers or fetchers are running, it will turn on one of each for you before it start.
• To see and modify the settings of what to crawl click the Options link next to start new crawl.
• Once you have started a crawl, Yioop will keep crawling until it runs out of new URLs to crawl and it times out or until you click Stop Crawl.
• From the time you click Stop Crawl until your crawl is ready is not instantaneous so please be patient... Do not stop the queue servers or fetchers. When Yioop has properly closed the active crawl, it will appear in the Previous Crawls list on the manage crawls page. At this point it should be safe to stop the fetchers and queue servers, if you want.

Seeing Search Results

• Pick a crawl that is listed under Previous Crawls and click on its "Set as Index" link.
• Click on the Yioop logo.
• This should take you back to the Yioop search screen.
• Typing in a query on this screen should now yield results from the crawl you set as index.

Quiz

Which of the following is true?

1. PDF documents are collections of compressed XML files.
2. Smoothing a first order language model by a zero'th order model allows us to gracefully handle plausible phrases that do not occur in a corpus, but might have occurred if the corpus were larger.
3. nextPhrase is a primitive method of our Inverted Index ADT.

Getting Help Within The App

• Yioop comes with an integrated wiki-based help system.
• Next to various widgets in the interface there is a question mark icon:
• Clicking on such an icon will bring up help about the adjacent widget. This might look like:
• Notice the help box has an edit link. This can be used to go to Yioop's wiki system and edit the help box if it needs to be improved.

Media Updater

• So far we've seen that Yioop has the main processes:
  1. Name Server - This is essentially the same as the Web App below if Yioop is running on at a single url (usually a different urls corresponds to different machines). If Yioop is run on multiple urls, one url will be configured to be a Name Server, and it is used for global messaging of information like the current crawls.
  2. Web App - this runs under a web server and is used to display search results, as well as user groups and wiki's. It also is used for sending messages between other processes.
  3. Queue Server - this consists of two sub-processes: (a) an indexer used to compute an inverted index from downloaded/imported documents, (b) a priority question qhich handles what documents to process next.
  4. Fetcher - get fetch batches from queue servers and then does initial downloading and processing of document before sending the result back to the appropriate queue server.
• In addition to crawling, there are many ongoing tasks that a web site might need to do for maintainence: compute usage analytics, bulk emailing, updating news and podcast feeds as well as trending information, compute group and feed item recommendations for users, convert uploaded media content to common media formats (convert to mp4, strip exif from images, etc).
• Yioop handles these tasks in with its Media Updater process which can be managed under Machines:
  
• The MediaUpdater only runs when it is turned on.

Controlling Jobs of the Media Updater

• Clicking the [Configure Media Jobs] link on the Manage Machines page takes one to a page:
  
• As can be seen by the [Name Server|Distributed] toggle, MediaUpdater jobs can be configure to run either only on the name server or across all machines in a Yioop site.
• The list beneath this shows a list of found media jobs for the MediaUpdater, and whether the job is active (On) or not (Off).
• To determine the Job list, Yioop looks in the folders: src/library/media_jobs (built-in) and work_directory/app/library/media_jobs (you can add new jobs here) for files with the suffice Jobs.php.
• We now focus on the FeedsUpdate Job which is responsible for downloading news feed items periodically for news searches.

Search Sources

• The Search Sources Activity can be used to add, edit, test, and delete feed search sources that the FeedsUpdate Job downloads each hour to look for feed items.
  
• Clicking the add button next to Media Sources lets you fill in a form to add a new source:
• The Media Kind drop down lets you choose which of several common news feed formats the to-be-downloaded-periodically url is. These include RSS/ATOM, JSON, scraped using XPaths from and HTML document, scraped using regex formats.
• The default Yioop install comes with several example feeds.
• The feeds currently added can be tested for their behavior using their Test links.