Yioop

Yioop is an open-source PHP search engine/indexer/crawler/IR library/wiki discussion board/ad platform created by me and some of master's students beginning in the Fall of 2009.
Details of when different features were added can be found in the Changelog.
Yioop can be downloaded from: https://www.seekquarry.com/p/Downloads
Alternatively, you can get various versions from the ViewGit Web interface to the Yioop's git repository or by cloning the most recent version using:
```
git clone https://seekquarry.com/git/yioop.git
```
You probably want to do the latter for this class.
It is highly recommended to briefly read the online documentation for Yioop.
To run Yioop you need: PHP 8.1 or greater with at least PDO sqlite support and compiled with the cURL, multi-byte string, libxml, bcmath, and GD libraries.
Yioop comes with/makes use of a single file Node/React-esque PHP WebServer/Router which may be independently interesting or useful for other projects.

Yioop Prerequisites if Running Under an Existing Web Server

Yioop can also be run under an existing web server, in which case, that web server should enable URL rewriting.
The version of Apache and PHP that come built-in to a Mac already support this.

In Ubuntu Linux (tested 22.04LTS), the following sequence of commands will get you what you need:

 sudo apt install curl
 sudo apt install php-cli
 sudo apt install php-mbstring
 sudo apt install php-sqlite
 sudo apt install php-curl
 sudo apt install php-gd
 sudo apt install php-xml
 sudo apt install php-bcmath

Alternatively, on Windows/Mac/Linux you could install Xampp. Just make sure to enable the php_curl.dll or php_curl.so extension in the php.ini file.

Configuration

Move the downloaded version of Yioop under your web server's DocumentRoot.
Yioop has a subfolder called work_directory. Make sure Yioop has read/write access to this directory.
Now point your browser to the location on localhost where you put Yioop. With any luck you should see a screen like:
Congratulations, you have successfully installed Yioop!

Logging in

To make a web crawl you need to login at this point. Click on the hamburger menu and choose Signin.
The default username is root and the password field should be left blank.
You should now see the default admin page:

Activities

Things you can do are broken down into activities which are listed under the Hamburger menu:
On login, you are on the Manage Account activity page. Here you can change your password if you like.
On the Manage Account page there also icon based quick links to a variety of useful activities. For example, the buttons

lead respectively to the Manage Crawls, Manage Machines, Crawl Mixes, and Query Statistics activities.
Returning to the admin activities listed under the hamburger menu, the Configure activity allows you to turn on error reporting, query info, and unit test, so can be useful for debugging.
Under Server Settings, you can control if accounts other than the root account can be created.
Users can be managed under Manage Users.
What activities a user has access to is controled by what roles they have (by default Admin or User). Roles consist of collection of allowed activities and can be managed under Manage Roles.
You are now ready to crawl!

Scripts for Crawling

Crawling involves two scripts, a QueueServer script and a Fetcher script, which are located in Yioop's src/executables directory.
These scripts can be run either from within the web interface (requires that the web server has permission to schedule jobs) or from the command line.
From the web interface to manually run these scripts go to Manage Machines, then click the "On" button for each of them.
The speed of a crawl is determined by the number of web pages that can be simultaneously downloaded (adding more fetchers tends to increase this), how quickly the priority queue of what to crawl next can be updated, and how quickly the downloaded files can be indexed (the number of queue servers affects the latter two).
A rule of thumb is that the queue server operates the most efficient if it can run on two CPU cores to itself and a fetcher operates most efficiently if it get a CPU core to itself.
Under the Manage Machines activity you can modify the number of queue servers and fetcher one has for a crawl or you can leave them at the default settings by using the "+" button.
If you want to run these scripts from the command line, open two command shells and in each go to Yioop's src/executables directory.
In one type:
```
php Fetcher.php terminal
```
in the other type:
```
php QueueServer.php terminal
```
Be aware that for this to work the php command has to be in your path environment variable; otherwise, you need to give the full path to php.

Crawling Nice

When a crawler or a web browser requests a web page it sends User-Agent info.
This info can tell a site owner who to contact in case your crawler behaves badly.
You can set this information for your crawler under the Configure activity:

Performing a Crawl

Now go back to the admin panel of Yioop and click on Manage Crawls.
If you type in a name of a crawl and click Start New Crawl, Yioop will start crawling with whatever the current crawl settings are.
If no queue servers or fetchers are running, it will turn on one of each for you before it start.
To see and modify the settings of what to crawl click the Options link next to start new crawl.
The Options page largely controls what urls or documents will be crawled, to control how individual documents are processed click on the [Page Processing Options] link:
The Test Options tab here let's you test how a single page will be crawled.
Going back to crawling, once you have started a crawl, Yioop will keep crawling until it runs out of new URLs to crawl and it times out or until you click Stop Crawl.
From the time you click Stop Crawl until your crawl is ready is not instantaneous so please be patient... Do not stop the queue servers or fetchers. When Yioop has properly closed the active crawl, it will appear in the Previous Crawls list on the manage crawls page. At this point it should be safe to stop the fetchers and queue servers, if you want.

Seeing Search Results

Pick a crawl that is listed under Previous Crawls and click on its "Set as Index" link.
Click on the Yioop logo.
This should take you back to the Yioop search screen.
Typing in a query on this screen should now yield results from the crawl you set as index.

In-Class Exercise

Download and install Yioop.
Set up a crawl with only one seed site, a URL of your choice.
Perform a crawl until you have at least a couple pages downloaded.
Set this crawl as your index. Do a search for a term on that page, take a screenshot of the result. Resize this image so its not too giant and post it to the discussion board.
Please post your solutions to the Aug 30 In-Class Exercise Thread.

Yioop Users and Groups

Yioop comes with the ability to allow new users to join a Yioop instance.
It also supports the ability for users to create groups.
A Yioop group is a place where users can create wiki pages and where user's can have threaded discussions.
Wiki pages can be one of several types:
- Standard wiki pages (using something like Wikipedia's wiki syntax but extended to allow links with relationships, if you want to make genealogy graphs, and with the ability to create forms that can be used to populate csv files).
- Page and comment feeds
- Pages that alias other pages
- Media folder pages (a little like drop box but can also auto add descriptions to some file based on search sources like IMDB). This supports playing of video in a webpage, (how this class' videos are served).
- Presentation pages (for slides)
- URL Shorteners to external pages
- Pastebin pages

Yioop Users and Groups

Yioop comes with several built-in groups:
- Search names of wiki pages in this group are viewed as search terms, the page itself is used for callout information when a user searched that term. The discussion group system for this group is used to allow a user to edit search results for existing crawls.
- Public used to control any pages on the web site that do not require a login. Can be used to edit the landing page and for the site blog.
- Help used for the internal help system that is called when user's click on [?] bottons.
- Moderation posts which are flagged in User groups are sent to this group where they can be moderated.
- Personal Groups each user's account has a personal group, threaded messages in this are used to implement the DM system.

Important Subcomponents- Jobs, Advertising, Analytics, Localization

Yioop has a subsystem to allow users to create a run jobs of various types.
Common jobs are: news and trending feeds based on search sources, thumbnail generation for images and videos, recoding videos, sending bulk mail etc.
Yioop has an interface that allows user's create and bid on keyword advertisements.
For the latter subsystem to make sense, Yioop has a variety of means to visual the analytics of crawls, search results, and group activity.
Yioop also has a subsystem that makes it possible to localize Yioop to different languages via a GUI interface.

Learning to Crawl

Outline

Yioop

Yioop Prerequisites if Running Under an Existing Web Server

Configuration

Logging in

Activities

Scripts for Crawling

Crawling Nice

Performing a Crawl

Seeing Search Results

In-Class Exercise

Yioop Users and Groups

Yioop Users and Groups

Important Subcomponents- Jobs, Advertising, Analytics, Localization