Abstract:

Yioop is an open source search engine. It supports many languages and has abilities to process and analyze text in different languages. Currently, there is some support for Chinese language processing in Yioop, such as Chinese text segmentation. However, the support for Chinese is limited and not very good. In this project, I am going to implement a better algorithm to segment Chinese text; I will implement an algorithm to do the part-of-speech tagging; and, I will also implement the Chinese question and answering system.

CS297 Results

• Implemented a dictionary based word suggestion system in Yioop.
• Implemented a Chinese Word segmentation Algorithm
• Partially implemented a Chinese part-of-speech tagging system
• Partially implemented a Chinese question and answering system

Proposed Schedule

Key Deliverables:

• Software
  • Enhance the Chinese Word segmentation Algorithm with memory optimization
  • Enhance the Chinese POS tagging Algorithm with trained weight
  • Implement a Chinese Question and Answering System
  • Implement a Chinese Entity Recognition System
• Report
  • CS 298 Report
  • CS 298 Presentation

Innovations and Challenges

• In general, the area of Natural Language Processing is a combination of many different areas such as linguistics, computer science, information engineering, and artificial intelligence.
• One of the most challenging parts is to implement Machine learning algorithms. The project needs to be implemented without external libraries. But, most of the existing works/papers are based on some very complicated machine learning models. So, I have to judge which models to use. It should be implementable as well as the results should look good.
• Since the project is open source, there are some constraints. One of them is the limit usage of memory. The source code is written in PHP. And, PHP arrays consume a lot of memory because it is a dynamic language. I implemented cache so that not everything would be loaded into memory.
• Different languages have different grammars. Chinese Sentences are much harder to deal with compared to English because they are less rigorous. Several Chinese sentences can be concatenated together with comma between them, which made them harder to seperate or to find the real subjective.

References: