Yioop

Yioop is an open-source PHP search engine/indexer/crawler/IR library/wiki discussion board/ad platform created by me beginning in the Fall of 2009.
Details of when different features were added can be found in the Changelog.
Yioop can be downloaded from: https://www.seekquarry.com/p/Downloads
Alternatively, you can get various versions from the ViewGit Web interface to the Yioop's git repository or by cloning the most recent version using:
```
git clone https://seekquarry.com/git/yioop.git
```
It is highly recommended to briefly read the online documentation for Yioop.
To run Yioop you need: PHP 5.6 or greater with at least PDO sqlite support and compiled with the cURL, multi-byte string, libxml, bcmath, and GD libraries.
Yioop comes with/makes use of a single file Node/React-esque PHP WebServer/Router which may be independently interesting or useful for other projects.

Yioop Prerequisites if Running Under an Existing Web Server

Yioop can also be run under an existing web server, in which case, that web server should enable URL rewriting.
The version of Apache and PHP that come built-in to a Mac already support this.

In Ubuntu Linux (tested 18.04LTS), the following sequence of commands will get you what you need:

 sudo apt install curl
 sudo apt install php7.2-cli
 sudo apt install php7.2-mbstring
 sudo apt install php7.2-sqlite
 sudo apt install php7.2-curl
 sudo apt install php7.2-gd
 sudo apt install php7.2-xml
 sudo apt install php7.2-bcmath

Alternatively, on Windows/Mac/Linux you could install Xampp. Just make sure to enable the php_curl.dll or php_curl.so extension in the php.ini file.

Configuration

Move the downloaded version of Yioop under your web server's DocumentRoot.
Yioop has a subfolder called work_directory. Make sure Yioop has read/write access to this directory.
Now point your browser to the location on localhost where you put Yioop. With any luck you should see a screen like:
Congratulations, you have successfully installed Yioop!
To make a web crawl you need to login at this point. The default username is root and the password field should be left blank.
You should now see the default admin page:
Things you can do are broken down into activities which are listed on the left hand side of the page.
On login, you are on the Manage Account activity page. Here you can change your password if you like.
You are now ready to crawl!

Scripts for Crawling

Crawling involves two scripts, a QueueServer script and a Fetcher script, which are located in Yioop's src/executables directory.
These scripts can be run either from within the web interface (requires that the web server has permission to schedule jobs) or from the command line.
From the web interface to manually run these scripts go to Manage Machines, then click the "On" button for each of them.
The speed of a crawl is determined by the number of web pages that can be simultaneously downloaded (adding more fetchers tends to increase this), how quickly the priority queue of what to crawl next can be updated, and how quickly the downloaded files can be indexed (the number of queue servers affects the latter two).
A rule of thumb is that the queue server operates the most efficient if it can run on two CPU cores to itself and a fetcher operates most efficiently if it get a CPU core to itself.
Under the Manage Machines activity you can modify the number of queue servers and fetcher one has for a crawl or you can leave them at the default settings.
If you want to run these scripts from the command line, open two command shells and in each go to Yioop's src/executables directory.
In one type:
```
php Fetcher.php terminal
```
in the other type:
```
php QueueServer.php terminal
```
Be aware that for this to work the php command has to be in your path environment variable; otherwise, you need to give the full path to php.

Crawling Nice

When a crawler or a web browser requests a web page it sends User-Agent info.
This info can tell a site owner who to contact in case your crawler behaves badly.
You can set this information for your crawler under the Configure activity:

Performing a Crawl

Now go back to the admin panel of Yioop and click on Manage Crawls.
If you type in a name of a crawl and click Start New Crawl, Yioop will start crawling with whatever the current crawl settings are.
If no queue servers or fetchers are running, it will turn on one of each for you before it start.
To see and modify the settings of what to crawl click the Options link next to start new crawl.
Once you have started a crawl, Yioop will keep crawling until it runs out of new URLs to crawl and it times out or until you click Stop Crawl.
From the time you click Stop Crawl until your crawl is ready is not instantaneous so please be patient... Do not stop the queue servers or fetchers. When Yioop has properly closed the active crawl, it will appear in the Previous Crawls list on the manage crawls page. At this point it should be safe to stop the fetchers and queue servers, if you want.

Seeing Search Results

Pick a crawl that is listed under Previous Crawls and click on its "Set as Index" link.
Click on the Yioop logo.
This should take you back to the Yioop search screen.
Typing in a query on this screen should now yield results from the crawl you set as index.

In-Class Exercise

Download a copy of Yioop by clicking on a ZIP link in the Yioop ViewGit Page.
Start Yioop running
Configure Yioop so the seed site consists of just your favorite Wikipedia page.
Start a crawl. What url does Yioop download before your Wikipedia page (look at the the fetcher's log to figure this out)?
Post your Wikipedia page url and the url Yioop downloaded first to the Sep 4 In-Class Exercise Thread.

Getting Help Within The App

Yioop comes with an integrated wiki-based help system.
Next to various widgets in the interface there is a question mark icon:
Clicking on such an icon will bring up help about the adjacent widget. This might look like:
Notice the help box has an edit link. This can be used to go to Yioop's wiki system and edit the help box if it needs to be improved.

Learning to Crawl

Outline

Yioop

Yioop Prerequisites if Running Under an Existing Web Server

Configuration

Scripts for Crawling

Crawling Nice

Performing a Crawl

Seeing Search Results

In-Class Exercise

Getting Help Within The App