CS297-298 Project News Feed

I have completed the first draft of my CS298 Final Report 
and am hoping to provide you with the same by Friday.

I was hoping that after you complete review of the same,
 and we finalise the changes needed to be made on the report
 the coming Tuesday (April 3rd), 

I could submit the same to the Computer Science department 
that week itself and the provide a demo and presentation to the 
committee after first showing you the same the very next week 
(April 10th ). 
 
I was hoping to defend my thesis on April 30th or May 1st or 2nd   

Tasks Completed:

Code :

Implementation of the code needed to implement the design.
These have been presented in Professor Chris Pollett's format 
and updated to the website.

Statistics :

On the basis of the guidance provided by Professor Pollett,
the following graphs have been plotted.

1> Music - Popularity
2> Difference between frequency prior to our implementation and 
after the implementation.
3> Value of music
4> Blob's accessed by each voter
5> Currency needed to be spend by each voter if he purchases all 
the music which he likes

On the basis of this statistics it has been observed that there 
is a considerable amount of improvement after our new logic has been implemented.

User#  OLD_FREQ	NEW_FREQ
1	2	2
2	36	0
3	1	1
5	10	2
6	0	0
7	28	4
8	0	2
9	2	0
10	6	6
11	10	10   

1> Submit CS298 Proposal in html format
Status : Completed 
would be submitted during meet today

2> Perform Deliverable 6: 
Implement Byzantine on IP address model.
Status : Complete
Would demo the same to Professor today. 
Need to perform some more brush ups before uploading it to the website.

3> Statistics of token ring model

Status : Partially complete

Have decided on the algorithm required to implement the same.
Would complete the same and present it to Professor
by next week meet.   

On the basis of the discussions (please refer previous blog)
the following deliverables need to be modified and submitted.

1. CS298 Proposal
2. CS297 Final Report
3. Deliverable 4 in uploadable format
4. Modifications to the program to complete deliverable 5
5. Start writing CS298 Final Report
6. Provide statistics for various scnearios on token ring network.

Meeting of February 13th was extremely fruitful in that 
we have been able to define our project with the scenario of zune devices
and also establish the goal of our project

The CS298 project has been defined as follows:

For ease in understanding let us take a scenario of
 multiple Zune devices within a network 
i.e.; they can communicate with each other and exchange music. 
There are two alternatives to obtain music; 
they can purchase music or they could squirt the same from any other zune device in their network. 

However, there is one limitation when you squirt the music (data), 
users can play it only a limited number of times 
and after that they need to squirt the music again. 
Alternatively some users may even choose to just purchase their favorite music. 

This results in two major issues coming forth
1.	There is a considerable amount of Bandwidth consumed each time
2.	Permanent copy of music is created on the new machine, 
thus resulting in wastage of memory.
The above discussed factors have a major impact especially 
when the two devices communicating with each other are located at a distance from each other.
The goal of our CS298 is to reduce the same.
This can be achieved by replicating a copy of this data 
on some other zune device which not only is also looking for the same data 
but also is located at a distance farther away from the original device,
thus ensuring that users on both side of the hemisphere have a 
copy which can be easily accessed by them. 

We would attempt to implement this on two difference network models

1.	Token Ring Network

2.	Internet Protocol (IP) Model 

Deliverable 4 implements the above on a token ring network.
The same has been completed and demonstrated to Professor Pollett.

1. CS298 Proposal
-----------------------

The CS298 proposal has been completed and the same has been reviewed by Professor Pollett.
 It has been submitted to the committee for further approval.


2. Deliverable 4
-------------------

This is the first deliverable for this semister.
 On te basis of the suggestions provided by professor Pollett during the previous semister,
 for this deliverable, implementation has been performed.

Would demonstrate the same to professor Pollett tommorow.

Brief description about deliverable 4

Deliverable 4 : Determine frequency of requests to BLOBs 
Description:
 
Our previous project implemented a real world scenario to implement a Distributed Byzantine implementation 
where all the nodes in question arrived at a decision for a Boolean valued data
 and also determined the most suitable machine in which to replicate a BLOB.
 Thus at the end of the previous deliverable we were able to create a model for distributed Byzantine agreement.
 This deliverable ntrates on moving this a step further.
 Here an attempt is made to determine which BLOBs are more commonly in demand.
 Voters will keep voting and depending on the BLOB they vote for determination is made regarding which BLOB is more commonly required. 
We could do this for certain time druation or for certain rounds (like in our example where we are performing this for 100 rounds) 
Consider that there are a set of BLOBS BSET = {B1, B2, B3, B4, B5, B6} 
and a and a set of Voters VSET = {V1, V2, V3, V4 } each with partial interest in some of the blobs B.
 For 100 rounds, on a random basis these voters cast votes. Depending on the votes cast counter for the votes is incremented.
 The voters along with their respective counter values are placed in a sorted list.
 Since java does not provide an easier approach to sort data, we sort the data using TreeMap.
 Criteria for choosing the BLOBs which need to be broadcasted has been chosen as 
The Top 2 Voters from the list 
or
BLOBs being requested more than 40% of the time

Once it has been decided as to which are the BLOB elements which have a higher frequency list , 
Byzantine Agreement is performed only for those BLOBs and the machine from which these BLOBs 
need to be extracted is obtained. 


 

Implement a program which checks frequency of requests for BLOBs. 

Mechanism such as setting timer/counters can be used to check  when each BLOB has been 
accessed and the duration for which the BLOBs have to wait.

In our current program for each BLOB we obtain information. 
Instead of this, we'll set up one or 2 BLOBs as the frequenct used BLOB.
Whenever a call for info is made to the database, we woudl also increment a counter.
Instead of just a counter we could use some of database features like sequence or triggers which would make this an 
automated process instead of  having it as a part of the code base.   

After all the work done last semister decided to take a week off and a small break.

Back to work, the deliverables which need to be dealt with through the next semister have 
been chalked as follows:

Deliverable 1:
---------------

Implement a program for frequency measurement of BLOB across a distributed network.

Deliverable 2:
----------------

Implement a program to create a composite set. 
Solution to the first problem of replication, which data needs to be replicated??

Deliverable 3:
---------------

Implement a program to improve the algorithm for partial broadcast / replication of data.

Deliverable 4:
----------------

Creation of testing framework and perform testing on the implementation performed.

Deliverable 5:
----------------

CS298 Final Project Report.

1.	Deliverable 3
Status: Complete
Modified the same according to professor pollett deliverable template and 
would provide the same today to professor for uploading to the website 

2.	Final Report – First draft
Status: Complete
Depending on the format suggested by professor Pollett , a detailed 30 page 
report has been prepared. Would go over the same with professor Pollett 
today. Modifications would be performed depending on the suggestions 
provided.

3.	Research on MANET – Final modifications to power point presentation
Status: Complete
Would submit the same to professor pollett today

4.	Dynamic XML documents with Distribution and Replication
Status: Completed
Would submit the same to professor pollett today

5.	Forming committee
Status: Partially complete
Professor Robert Chun has accepted to be a part of the committee

6.	CS298 Proposal First draft
Status: Partially complete
Have prepared a rough draft based on my understanding of the problem, 
However need to discuss things further with professor Pollett also need to 
do more research on the idea and form a proper concrete understanding of the 
problem.

Once this has been performed, the CS298 proposal would be further modified

7.	BLOG
Status: Complete

All my BLOG entries so far exceed 80 columns. Have changed all the entries 
to be less than 80 columns for ease in reading.


   

1. Algorithm

Status : Completed
deliverable 3 has been completed and a demo for the same provided to Prof 
Pollett

Need to create deliverable 3 according to professor's template for updating the same to the website.

2. Research on Manet

Status: Complete
Approximately 10-15 different papers have been read on this particular topic. 
Have prepared a power point presentation based on the same. Professor Pollett 
has approved the same with some minor modifications. Once this is done it would 
be uploaded to the website.

3. Dynamic XML documents with Distribution and Replication

Status: Complete

Professor Pollett has approved the same with some minor modifications. Once this 
is done it would be uploaded to the website.

4. Event Points Annotating XML Documents for Remote Sharing

Status: Complete

This does not have too much relevance with our area of research. Hence this 
would not be uploaded.

5. Google Videos

Status: Complete

Information present in the power point presentation would not have too much 
relevance with our area of research. Hence this would not be uploaded.

   

1> Deliverable 3: Voter's Algorithm
---------------------------------
Modify deliverable 2(Byzantine Agreement) as the follows
a)
Deliverable 2, had two votes which could be provided by voters 
(heads or tails), in this deliverable 3, we are going to consider 
the votes to be the machine names where the BLOB data could be stored.
Initially it would be assumed that we have 8 machines and later on 
We would try to make it dynamic by counting the number of machines 
present form parsing the data from the XML document.

b) Parse the XML document and obtain the location from the same. This 
location is the parameter based on which all the votes would be casted

c) Earlier voters could be of 2 types, faulty or non faulty. In this 
new scenario we would have 3 categories into which the voters can fall 
into; faulty or non faulty or people who don't have/need that 
particular BLOB object.

The 3rd category is the newly added category, here we would submit 
random votes initially and once all the non faulty votes come on a 
decision , these voters would also vote the same as the decision, so 
that a Byzantine agreement can be reached.

d) Modify the XML document such that there are multiple location 
parameters present in each document.   

This is not a priority item and would be performed only after a) b) and 
c).   

2> Research

Try to do some research on T-mobile network . maybe something like 
where all the commiting data back to one of the several central 
database.Do some search on "Manet"


3> Try to do more research on Google Videos

4> "Dynamic XML documents with Distribution and Replication"

Read and Understood and have prepared a Power point presentation  on 
the same. Would demonstrate the same to professor Pollett today.

5> "Event Points" Annotating XML Documents for Remote Sharing

Read and Understood and have prepared a Power point presentation  on 
the same. Would demonstrate the same to professor Pollett today.

   

1> Byzantine Agreement
----------------------

Status : Completed

Byzantine agreement has been completed sucessfully.

For our Byzantine example, we have 2 machines, 8 voters (4 each on each 
machine), 2 faulty voters (1 on each machine).

In order to ensure flexibility care has been taken while writing the 
code to ensure that it is possible to extend this program to any number 
of machines simply by adding the ip of the same to Manager.java and 
copying all the code to the other machine.

RMI has been used inorder to be able to implement distributed 
functionality. By using RMI the voters present in different machine 
communicate with each other, exactly in the same fashion as they would 
have if they were located on the same machine.

JDBC features are used to store the votes obtained by each voter to the 
database.

Deliverable 2.html has been completed and would be provided to 
professor Chris Pollett after demonstration of the same for uploading 
to the website.

2>  "Database Replication techniques: a three parameter classification"

Status : Completed

Suggestions provided by  Professor Pollett have been applied to the 
presentation and would be submitted today.

3> "Dynamic XML documents with Distribution and Replication"

Status : Completed

Read and Understood and have prepared a Power point presentation  on 
the same. Would demonstrate the same to professor Pollett today.

4> "Event Points" Annotating XML Documents for Remote Sharing

Status: Completed

Read and Understood and have prepared a Power point presentation  on 
the same. Would demonstrate the same to professor Pollett today.

5> Find out more information on how google streams video

Status: Partially complete

Have obtained certain information and run the same over to Professor 
Pollett.

However, Professor Pollett has requested to some more information 
regarding few specific topics present in the slides. Research on the 
same is in progress.

   

Byzantine Agreement


Algorithm for the same has been created. Have started writing a small 
program which performs the task of ElectionManager.java and Voter.java.

One of the most important tasks this week was to try to get JDBC. 
Basically there were multiple versions of postgreSql present (the 
default one which comes with Linux and SonyX - which had been 
installed). Had to uninstall and reinstall postgres on both the 
computers and now able to run a small program which connects to the 
database and populates values correctly.

The same principle needs to be incorporated into our project right now.

   

Discussions performed/ Future tasks
----------------------------------

1.Presentation on Distributed Algorithms

Status : Completed


Modifications suggested by professor Pollett completed and the same 
would be submitted. today

2. Perform additional modification to deliverable 1 (a) xml file name 
should not be hardcoded  b) change it to the format requried for 
submission) 

Status : Completed


All the required changes suggested by professor Pollett has been 
performed and the same would be delivered to Professor Pollett today.

3. Read, understand and present the paper "Database Replication techniques: a three parameter classification"

Status : Completed

The same has been completed, would run over the same today with 
Professor Pollett.

4. Find out more information on how google streams video

Partially complete

Have obtained certain information and run the same over to Professor 
Pollett.
However, Professor Pollett has requested to some more information 
regarding few specific topics present in the slides. Research on the 
same is in progress.

5. Byzantine Agreement

Have read and understood Byzantine Agreement. Also have read 
information about Java sockets.

An algorithm and design have been prepared for the same. Would run the 
same over with Professor Pollett. Implementation has commenced and 
attempt woudl be made to complete the same before next meeting.

Also a powerpoint presentation, which explains all the fundamentals 
regarding Byzantine agreement and our apprach towards it has also been 
prepared.

   

Status of tasks assigned
----------------------------

1: “Complete modifications to presentation II, on the basis of the 
suggestions provided by professor Pollett.”

Status : Completed and uploaded

2.	Complete whatever is pending from last week task 

2: “Perform research on how google streams video and create a 
presentation on the same.”

Status : Completed and Presented the same. Professor Pollett has 
suggested few more things on which he would like more information to be 
made available, need to perform the same.

3.	Change the implementation of Byzantine’s agreement algorithm 
which has been mentioned in the Homework. Our implementation needs to 
be changed to 2 databases; each has a copy of the dumb table with a 
vote in it; you are making connection to the database to 
synchronization. Provide Professor Pollett with a research on how you 
would modify and what logic you would use to implement the same.

4.	Complete whatever is pending from last week task 3: “Solve all 
the issues which are being faced with respect to Deliverable 1 and 
provide Professor Pollett with a demonstration and submit the same.”

Status : The same has been completed and presented.  Professor Pollett 
has requested an additional modification which needs to be performed.


5. Byzantine Agreement

Still reading up materials on the same, need to try to perform socket 
programing and come out with a technique to perform the same.

Discussions performed/ Future tasks
----------------------------------

1. Find out more information on how google streams video

2. Perform additional modification to deliverable 1 (a) xml file name 
should not be hardcoded  b) change it to the format requried for 
submission) 

Byzantine Agreement

3. Still reading up materials on the same, need to try to perform 
socket programing and come out with a technique to perform the same.

4. Read paper 1 and provide a power point presentation on the same

   

Activities to be completed by October 10th

1.	Complete modifications to presentation II, on the basis of the 
suggestions provided by professor Pollett.

Status : Completed

The same would be submitted to professor Pollett during the meeting 
next week.

2.	Perform research on how google streams video and create a presentation on the same.

Status : Completed

Have been able to get some infomation though not extensive information. 

Have prepared a small power point presentation. Would discuss the same 
with Professor Pollett during meet next week.

3.	Complete whatever is pending from last week task 3: Solve all 
the issues which are being faced with respect to Deliverable 1 and 
provide Professor Pollett with a demonstration and submit the same.

Status : Completed
Problems with respect to deliverable 1 have been taken care of and 
correct output for the same is observed.

Would demonstrate the same to Professor Pollett, during next meet.

   

Activities to be completed by October 10th

1.	Complete modifications to presentation II, on the basis of the 
suggestions provided by professor Pollett.
2.	Perform research on how google streams video and create a 
presentation on the same.
3.	Complete whatever is pending from last week task 3: Solve all 
the issues which are being faced with respect to Deliverable 1 and 
provide Professor Pollett with a demonstration and submit the same.
 
Activities to be completed by October 17th

1.	Complete whatever is pending from last week task 1: “Complete 
modifications to presentation II, on the basis of the suggestions 
provided by professor Pollett.”
2.	Complete whatever is pending from last week task 2: “Perform 
research on how google streams video and create a presentation on the 
same.”
3.	Change the implementation of Byzantine’s agreement algorithm 
which has been mentioned in the Homework. Our implementation needs to 
be changed to 2 databases; each has a copy of the dumb table with a 
vote in it; you are making connection to the database to 
synchronization. Provide Professor Pollett with a research on how you 
would modify and what logic you would use to implement the same.
4.	Complete whatever is pending from last week task 3: “Solve all 
the issues which are being faced with respect to Deliverable 1 and 
provide Professor Pollett with a demonstration and submit the same.”

Concepts Discussed:

Concept 1: 
	Global Data Dictionary, needs to be replicated. We could look at 
this aspect I 2 angles;
1.	Data dictionary consists of xml data.
2.	Could be stored in xml structure, which consists of some BLOB 
objects.

We need to devise an algorithm to do so;

Status of tasks due on 
September 19th 2006:

1>	Read the following chapters from "Principles of Distributed Database System".
 Make Power point presentations on the same.

a> Chapter 12: Distributed DBMS Reliability

Status : Completed
The chapter has been read, understood and powerpoint slide of the same 
completed.

b> Chapter 14: Distributed Object Database Management Systems

Status : Completed
The chapter has been read, understood and powerpoint slide of the same 
completed.

c> Chapter 16: Current Issues

Status : Completed
The chapter has been read, understood and powerpoint slide of the same 
completed.

2> Submit CS297 Proposal (in Professor Chris Pollett's template).

Status : Completed

3> Submit a brief Bio to be put up on the website.

Status : Completed

4> Complete whatever is remaining of tasks 4 and 5 from previous week.

Status : In Progress

5> Get postgres setup on Linux

Status : Completed

Activities to be completed by September 26th 2006

1> Perform modifications on power point slides (prepared for chapter 
4,5 and 7.5) according to suggestions provided by Professor Chris 
Pollett.

2> Create a function in postgres, the function implementation would be 
in C/Java programming language. Take data from an Xml file (maybe 
something like an address) and shred it into the postgres database.

Activities to be completed by October 3rd 2006

Read,understand and perform powerpoint slides for the following 
chapters from the book "Distributed Algorithms"

1> Chapter 6: Distributed Consensus with Process Failures

2> Chapter 7: More consensus problems

3> Chapter 8: Modelling II: Asynchronous System Model

1> Read the following chapters from "Principles of Distributed Database 
System". Make Power point presentations on the same.

a> Chapter 4: Distributed DBMS Architecture
b> Chapter 5: Distributed Database Design
c> Section 7.5: Layers of Query Processing

Status : Completed

All the above 3 chapters (total of 89 pages) have been read understood 
and a powerpoint presentation providing an explanation of he concepts 
has also bee prepared for the same.

2> Install Linux on the Home Desktop

Status : Completed

Installation of Red Hat Linux version 9 has been succesfully completed 
on the desktop

3> Install Dynamic DNS Server, so that you can access your desktop from 
SJSU.

Status : Completed

Dynamic DNS has been succesfully setup on the desktop and testing on 
the same has also been performed by trying to connect to it from IBM.

The Dynamic DNS has been installed from www.dynamicdns.com

The same can be accessed by using http://preethi.homelinux.net/

4. Get postgres setup on Linux

Status : In progress

Have performed research on trying to find a distributed postgres which 
would match our requirement.

Currently trying to install "Slony-I --- A replication system for 
PostgreSQL "

Slony-I is a "master to multiple slaves" replication system with 
cascading and failover. The big picture for the development of Slony-I 
is a master-slave system that includes all features and capabilities 
needed to replicate large databases to a reasonably limited number of 
slave systems. 

More details about the same can be observed in 
"http://gborg.postgresql.org/project/slony1/projdisplay.php"

5> Try to add some Create Function

Status : Incomplete

This activity (which is deliverable 1 due on Sept 30th), cannot be 
performed till task 4 is complete.

   

Tasks to be completed by September 12th 2005 :

1> Read the following chapters from "Principles of Distributed Database 
System". Make Power point presentations on the same.

a> Chapter 4: Distributed DBMS Architecture
b> Chapter 5: Distributed Database Design
c> Section 7.5: Layers of Query Processing
2> Install Linux on the Home Desktop
3> Install Dynamic DNS Server, so that you can access your desktop from SJSU.
4> Get postgres setup on Linux
5> Try to add some Create Function

Tasks to be completed by September 24th 2005 :

1> Read the following chapters from "Principles of Distributed Database 
System". Make Power point presentations on the same.

a> Chapter 12: Distributed DBMS Reliability
b> Chapter 14: Distributed Object Database Management Systems
c> Chapter 16: Current Issues
2> Submit CS297 Proposal (in Professor Chris Pollett's template).
3> Submit a brief Bio to be put up on the website.
4> Complete whatever is remaining of tasks 4 and 5 from previous week.