Yioop

Yioop is an open-source PHP search engine/indexer/crawler/IR library/wiki discussion board/ad platform created by me beginning in the Fall of 2009.
It can be downloaded from: https://www.seekquarry.com/p/Downloads
Alternatively, you can get various versions from the ViewGit Web interface to the Yioop's git repository or by cloning the most recent version using:
```
git clone https://seekquarry.com/git/yioop.git
```
It is highly recommended to briefly read the online documentation for Yioop.
To run Yioop you need: PHP 5.6 or greater with at least PDO sqlite support and compiled with the cURL, multi-byte string, libxml, bcmath, and GD libraries.
Yioop comes with/makes use of a single file Node/React-esque PHP WebServer/Router which may be independently interesting or useful for other projects.

Yioop Prerequisites if Running Under an Existing Web Server

Yioop can also be run under an existing web server, in which case, that web server should enable URL rewriting.
The version of Apache and PHP that come built-in to a Mac already support this.

In Ubuntu Linux (tested 18.04LTS), the following sequence of commands will get you what you need:

sudo apt install curl
 sudo apt install php7.2-cli
 sudo apt install php7.2-mbstring
 sudo apt install php7.2-sqlite
 sudo apt install php7.2-curl
 sudo apt install php7.2-gd
 sudo apt install php7.2-xml
 sudo apt install php7.2-bcmath

Alternatively, on Windows/Mac/Linux you could install Xampp. Just make sure to enable the php_curl.dll or php_curl.so extension in the php.ini file.

Configuration

Move the downloaded version of Yioop under your web server's DocumentRoot.
Yioop has a subfolder called work_directory. Make sure Yioop has read/write access to this directory.
Now point your browser to the location on localhost where you put Yioop. With any luck you should see a screen like:
Congratulations, you have successfully installed Yioop!
To make a web crawl you need to login at this point. The default username is root and the password field should be left blank.
You should now see the default admin page:
Things you can do are broken down into activities which are listed on the left hand side of the page.
On login, you are on the Manage Account activity page. Here you can change your password if you like.
You are now ready to crawl!

Scripts for Crawling

Crawling involves two scripts, a QueueServer script and a Fetcher script, which are located in Yioop's src/executables directory.
These scripts can be run either from within the web interface (requires that the web server has permission to schedule jobs) or from the command line.
From the web interface to manually run these scripts go to Manage Machines, then click the "On" button for each of them.
The speed of a crawl is determined by the number of web pages that can be simultaneously downloaded (adding more fetchers tends to increase this), how quickly the priority queue of what to crawl next can be updated, and how quickly the downloaded files can be indexed (the number of queue servers affects the latter two).
A rule of thumb is that the queue server operates the most efficient if it can run on two CPU cores to itself and a fetcher operates most efficiently if it get a CPU core to itself.
Under the Manage Machines activity you can modify the number of queue servers and fetcher one has for a crawl or you can leave them at the default settings.
If you want to run these scripts from the command line, open two command shells and in each go to Yioop's src/executables directory.
In one type:
```
php Fetcher.php terminal
```
in the other type:
```
php QueueServer.php terminal
```
Be aware that for this to work the php command has to be in your path environment variable; otherwise, you need to give the full path to php.

Crawling Nice

When a crawler or a web browser requests a web page it sends User-Agent info.
This info can tell a site owner who to contact in case your crawler behaves badly.
You can set this information for your crawler under the Configure activity:

Performing a Crawl

Now go back to the admin panel of Yioop and click on Manage Crawls.
If you type in a name of a crawl and click Start New Crawl, Yioop will start crawling with whatever the current crawl settings are.
If no queue servers or fetchers are running, it will turn on one of each for you before it start.
To see and modify the settings of what to crawl click the Options link next to start new crawl.
Once you have started a crawl, Yioop will keep crawling until it runs out of new URLs to crawl and it times out or until you click Stop Crawl.
From the time you click Stop Crawl until your crawl is ready is not instantaneous so please be patient... Do not stop the queue servers or fetchers. When Yioop has properly closed the active crawl, it will appear in the Previous Crawls list on the manage crawls page. At this point it should be safe to stop the fetchers and queue servers, if you want.

Seeing Search Results

Pick a crawl that is listed under Previous Crawls and click on its "Set as Index" link.
Click on the Yioop logo.
This should take you back to the Yioop search screen.
Typing in a query on this screen should now yield results from the crawl you set as index.

In-Class Exercise

Download a copy of Yioop by clicking on a ZIP link in the Yioop ViewGit Page.
Start Yioop running
Configure Yioop so the seed site consists of just your favorite Wikipedia page.
Start a crawl. What url does Yioop download before your Wikipedia page (look at the the fetcher's log to figure this out)?
Post your Wikipedia page url and the url Yioop downloaded first to the Sep 5 In-Class Exercise Thread.

Getting Help Within The App

Yioop comes with an integrated wiki-based help system.
Next to various widgets in the interface there is a question mark icon:
Clicking on such an icon will bring up help about the adjacent widget. This might look like:
Notice the help box has an edit link. This can be used to go to Yioop's wiki system and edit the help box if it needs to be improved.

Configuring Your Crawl

Yioop supports numerous options for controlling what sites you crawl and how you crawl them.
The relevant activities for controlling these can be found under the Crawls activities group:
- Manage Crawl is used for running crawls, managing existing crawls, and specifying what sites crawls are allowed to go to.
- Manage Classifiers is used to create web page classifiers using existing crawls, these classifiers can then be used to label web pages during new crawls
- Page Options is used to control what file types can be indexed and how data is extracted from a crawled page.
- Results Editor is used to control what sites are allowed to appear in search results from an existing crawl, for example, preventing a site from appearing after a take down notice.
- Search Sources is used to specify urls which should be treated as video pages as well as urls which should be periodically updated as news pages. Periodic updating of pages only occurs if the MediaUpdater process is turned on under Manage Machines.

Crawl Options

By default, Yioop crawls a collection of web sites I thought were popular.
If one goes to Manage Crawls and clicks the Options link next to the Start New Crawl button one should see:
The tabs at the top determine if Yioop will do a web crawl (default) or a local crawl of some text repositories such as a Wikipedia Dump.
Crawl Order specifies what kind of queue is used to decide what to crawl next. The two choices are Page Importance, or Breadth-first.
The Get Crawl Options From drop down lets you load option from previous crawls you have done.
Checking the Restrict Sites By Url checkbox will display another textarea in which you can enter the only allowed urls for the next crawl to download.
Disallowed Sites/Sites with Quotas disallowed sites can be used to specify pages that Yioop is not allowed to download or to rate limit some sites.
Seed Sites is the list of urls the crawl will start from.

Page Options

The Page Options activity has three tabs:
The Crawl Time tab is used to control the processing of documents at crawl time. For example: What file extensions will be processed at all. How many bytes to use/download from a document. What summarizer should be run on a document and how many bytes output is it allowed to produce. What classifiers should be used. How long between crawls of a page. What classifiers should be used to label or rank documents. Which indexing plugins should be used. What operations in the Page Rules language should be applied.
The Search Time tab is used to control at search time how search results are displayed.
The Test Options tab is used how the options set above effect the processing of a given page. It has a text area that you can cut and paste the source code of a page into, and then run and get results of how Yioop would process the page.

Other Things Yioop Can Do

Under the Social activities group:

are three activities used in managing and view groups of users within Yioop.
Admins and users can use Manage Groups to create and manage groups of other users at various access levels.
Each group comes with its own discussion Feed and Wiki.
On install, Yioop comes with two built in groups: Public and Help.
These are configured by default to be only editable by the admin account.
The Public wiki together with the Appearance activity in Yioop can be used to control the look of a Yioop web site. For example, the seekquarry.com website is a vanilla install of Yioop where this has been done to change its appearance.
The Configure activity is another useful activity. You can use it to turn on Query info and Error Info. These can be used to understand what iterators Yioop will use in processing a query.

Learning to Crawl

Outline

Yioop

Yioop Prerequisites if Running Under an Existing Web Server

Configuration

Scripts for Crawling

Crawling Nice

Performing a Crawl

Seeing Search Results

In-Class Exercise

Getting Help Within The App

Configuring Your Crawl

Crawl Options

Page Options

Other Things Yioop Can Do